2024-03-29T11:08:21.031 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-29T11:08:21.032 DEBUG:teuthology.run:Teuthology command: teuthology --name amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi --owner scheduled_amathuri@teuthology --archive /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622694 --description rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} --verbose -- /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622694/orig.config.yaml 2024-03-29T11:08:21.065 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:08:21.150 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622694 branch: main description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} email: null first_in_suite: false job_id: '7622694' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 10 - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: main ceph: conf: client: debug monc: 20 debug ms: 1 debug objecter: 20 debug rados: 20 global: lockdep: true mon client directed command retry: 5 mon election default strategy: 1 ms bind msgr2: false ms inject socket failures: 5000 ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 mon scrub interval: 300 mon warn on pool no app: false paxos service trim min: 10 osd: bluestore block size: 96636764160 bluestore compression algorithm: snappy bluestore compression mode: aggressive bluestore fsck on mount: true bluestore zero block detection: true debug bluefs: 20 debug bluestore: 20 debug monc: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd class default list: '*' osd class load list: '*' osd client watch timeout: 120 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd mclock override recovery settings: true osd mclock profile: high_recovery_ops osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random osd shutdown pgref assert: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME - reached quota - overall HEALTH_ - \(CACHE_POOL_NO_HIT_SET\) - \(POOL_FULL\) - \(SMALLER_PGP_NUM\) - \(SLOW_OPS\) - \(CACHE_POOL_NEAR_FULL\) - \(POOL_APP_NOT_ENABLED\) - \(PG_AVAILABILITY\) - \(OBJECT_MISPLACED\) - slow request mon_bind_msgr2: false sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: main sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 owner: scheduled_amathuri@teuthology priority: 150 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - node-exporter.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - node-exporter.b seed: 8152 sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 sleep_before_teardown: 0 subset: 99/1000 suite: rados:verify suite_branch: main suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1fb5d10608dccee73edd6ef8a2c7b804a097c324 targets: smithi086.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKxPlMYyYmBRZDXnysowdOFSkkk1PV8lkGFIQWavm/+mvY2O+h5sZqC/0bFk9+9k39a/tF2jq129pkZF9EhNJGE= smithi196.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBITeA4FH29GBMbjGs2YpZjmD02Lke+TSUSNBt/bSNE5Jj2yN5L+e0jVZLnzu8Z5jXcWb02DMbALytQ3wv0SU21k= tasks: - install: null - ceph: null - workunit: clients: client.0: - rados/test.sh env: ALLOW_TIMEOUTS: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-26_05:45:44 tube: smithi user: amathuri verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-29T11:08:21.150 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa; will attempt to use it 2024-03-29T11:08:21.151 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks 2024-03-29T11:08:21.151 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-29T11:08:21.153 INFO:teuthology.task.internal:Checking packages... 2024-03-29T11:08:21.174 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5dcb190da270440ef141d088eb27a9fdb8afc5f2' 2024-03-29T11:08:21.174 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-29T11:08:21.174 INFO:teuthology.packaging:ref: None 2024-03-29T11:08:21.174 INFO:teuthology.packaging:tag: None 2024-03-29T11:08:21.174 INFO:teuthology.packaging:branch: main 2024-03-29T11:08:21.174 INFO:teuthology.packaging:sha1: 5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:08:21.174 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=main 2024-03-29T11:08:21.450 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2530.gb74786fc 2024-03-29T11:08:21.452 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-29T11:08:21.459 INFO:teuthology.task.internal:no buildpackages task found 2024-03-29T11:08:21.459 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-29T11:08:21.495 INFO:teuthology.task.internal:Saving configuration 2024-03-29T11:08:21.513 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-29T11:08:21.520 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:08:21.542 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622694', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:09.230609', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKxPlMYyYmBRZDXnysowdOFSkkk1PV8lkGFIQWavm/+mvY2O+h5sZqC/0bFk9+9k39a/tF2jq129pkZF9EhNJGE='} 2024-03-29T11:08:21.564 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi196.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622694', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:09.229153', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBITeA4FH29GBMbjGs2YpZjmD02Lke+TSUSNBt/bSNE5Jj2yN5L+e0jVZLnzu8Z5jXcWb02DMbALytQ3wv0SU21k='} 2024-03-29T11:08:21.564 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-29T11:08:21.570 INFO:teuthology.task.internal:roles: ubuntu@smithi086.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'] 2024-03-29T11:08:21.570 INFO:teuthology.task.internal:roles: ubuntu@smithi196.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b'] 2024-03-29T11:08:21.570 INFO:teuthology.run_tasks:Running task console_log... 2024-03-29T11:08:21.657 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f976b3890d0>, signals=[15]) 2024-03-29T11:08:21.658 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-29T11:08:21.664 INFO:teuthology.task.internal:Opening connections... 2024-03-29T11:08:21.664 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi086.front.sepia.ceph.com 2024-03-29T11:08:21.666 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:08:21.764 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi196.front.sepia.ceph.com 2024-03-29T11:08:21.765 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:08:21.851 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-29T11:08:21.857 DEBUG:teuthology.orchestra.run.smithi086:> uname -m 2024-03-29T11:08:21.874 INFO:teuthology.orchestra.run.smithi086.stdout:x86_64 2024-03-29T11:08:21.874 DEBUG:teuthology.orchestra.run.smithi086:> cat /etc/os-release 2024-03-29T11:08:21.930 INFO:teuthology.orchestra.run.smithi086.stdout:NAME="CentOS Stream" 2024-03-29T11:08:21.930 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION="9" 2024-03-29T11:08:21.930 INFO:teuthology.orchestra.run.smithi086.stdout:ID="centos" 2024-03-29T11:08:21.930 INFO:teuthology.orchestra.run.smithi086.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:08:21.931 INFO:teuthology.orchestra.run.smithi086.stdout:VERSION_ID="9" 2024-03-29T11:08:21.931 INFO:teuthology.orchestra.run.smithi086.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:08:21.931 INFO:teuthology.orchestra.run.smithi086.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:08:21.931 INFO:teuthology.orchestra.run.smithi086.stdout:ANSI_COLOR="0;31" 2024-03-29T11:08:21.931 INFO:teuthology.orchestra.run.smithi086.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:08:21.931 INFO:teuthology.orchestra.run.smithi086.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:08:21.931 INFO:teuthology.orchestra.run.smithi086.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:08:21.931 INFO:teuthology.orchestra.run.smithi086.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:08:21.931 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:08:21.931 INFO:teuthology.orchestra.run.smithi086.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:08:21.932 INFO:teuthology.lock.ops:Updating smithi086.front.sepia.ceph.com on lock server 2024-03-29T11:08:21.950 DEBUG:teuthology.orchestra.run.smithi196:> uname -m 2024-03-29T11:08:21.968 INFO:teuthology.orchestra.run.smithi196.stdout:x86_64 2024-03-29T11:08:21.968 DEBUG:teuthology.orchestra.run.smithi196:> cat /etc/os-release 2024-03-29T11:08:22.024 INFO:teuthology.orchestra.run.smithi196.stdout:NAME="CentOS Stream" 2024-03-29T11:08:22.024 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION="9" 2024-03-29T11:08:22.024 INFO:teuthology.orchestra.run.smithi196.stdout:ID="centos" 2024-03-29T11:08:22.024 INFO:teuthology.orchestra.run.smithi196.stdout:ID_LIKE="rhel fedora" 2024-03-29T11:08:22.024 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_ID="9" 2024-03-29T11:08:22.024 INFO:teuthology.orchestra.run.smithi196.stdout:PLATFORM_ID="platform:el9" 2024-03-29T11:08:22.024 INFO:teuthology.orchestra.run.smithi196.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-29T11:08:22.024 INFO:teuthology.orchestra.run.smithi196.stdout:ANSI_COLOR="0;31" 2024-03-29T11:08:22.024 INFO:teuthology.orchestra.run.smithi196.stdout:LOGO="fedora-logo-icon" 2024-03-29T11:08:22.024 INFO:teuthology.orchestra.run.smithi196.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-29T11:08:22.024 INFO:teuthology.orchestra.run.smithi196.stdout:HOME_URL="https://centos.org/" 2024-03-29T11:08:22.024 INFO:teuthology.orchestra.run.smithi196.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-29T11:08:22.024 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-29T11:08:22.025 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-29T11:08:22.025 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server 2024-03-29T11:08:22.041 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-29T11:08:22.051 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-29T11:08:22.105 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-29T11:08:22.105 DEBUG:teuthology.orchestra.run.smithi086:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:08:22.108 DEBUG:teuthology.orchestra.run.smithi196:> test '!' -e /home/ubuntu/cephtest 2024-03-29T11:08:22.123 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-29T11:08:22.147 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-29T11:08:22.147 DEBUG:teuthology.orchestra.run.smithi086:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:08:22.165 DEBUG:teuthology.orchestra.run.smithi196:> test -z $(ls -A /var/lib/ceph) 2024-03-29T11:08:22.284 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-29T11:08:22.340 INFO:teuthology.run_tasks:Running task kernel... 2024-03-29T11:08:22.380 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-29T11:08:22.380 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-29T11:08:22.381 DEBUG:teuthology.orchestra.run.smithi086:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:08:22.381 DEBUG:teuthology.orchestra.run.smithi196:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-29T11:08:22.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:08:22.397 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2024-03-29T11:08:22.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:08:22.398 DEBUG:teuthology.orchestra.run.smithi196:> uname -r 2024-03-29T11:08:22.453 INFO:teuthology.orchestra.run.smithi086.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:08:22.454 INFO:teuthology.task.kernel:Running kernel on smithi086: 5.14.0-361.el9.x86_64 2024-03-29T11:08:22.454 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum install -y kernel 2024-03-29T11:08:22.456 INFO:teuthology.orchestra.run.smithi196.stdout:5.14.0-361.el9.x86_64 2024-03-29T11:08:22.456 INFO:teuthology.task.kernel:Running kernel on smithi196: 5.14.0-361.el9.x86_64 2024-03-29T11:08:22.456 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum install -y kernel 2024-03-29T11:08:24.655 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-29T11:08:24.894 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS-9 - AppStream 8.5 MB/s | 6.7 MB 00:00 2024-03-29T11:08:26.297 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-29T11:08:26.524 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS-9 - BaseOS 3.8 MB/s | 2.0 MB 00:00 2024-03-29T11:08:27.274 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - BaseOS 13 MB/s | 8.0 MB 00:00 2024-03-29T11:08:27.767 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - BaseOS 9.1 MB/s | 8.0 MB 00:00 2024-03-29T11:08:29.549 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - AppStream 27 MB/s | 19 MB 00:00 2024-03-29T11:08:34.249 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - CRB 4.9 MB/s | 6.1 MB 00:01 2024-03-29T11:08:35.901 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 16 kB 00:00 2024-03-29T11:08:37.176 INFO:teuthology.orchestra.run.smithi196.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-03-29T11:08:42.259 INFO:teuthology.orchestra.run.smithi196.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-29T11:08:44.482 INFO:teuthology.orchestra.run.smithi196.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:08:44.482 INFO:teuthology.orchestra.run.smithi196.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:08:44.583 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T11:08:44.595 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:08:44.595 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-29T11:08:44.595 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:08:44.595 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-29T11:08:44.595 INFO:teuthology.orchestra.run.smithi196.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:08:44.595 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-29T11:08:44.595 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:08:44.595 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:08:44.595 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:08:44.595 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:08:44.595 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-29T11:08:44.595 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:08:44.595 INFO:teuthology.orchestra.run.smithi196.stdout:Install 4 Packages 2024-03-29T11:08:44.596 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:08:44.596 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 76 M 2024-03-29T11:08:44.596 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 123 M 2024-03-29T11:08:44.596 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-29T11:08:44.802 INFO:teuthology.orchestra.run.smithi196.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 931 kB/s | 192 kB 00:00 2024-03-29T11:08:46.810 INFO:teuthology.orchestra.run.smithi196.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 6.8 MB/s | 15 MB 00:02 2024-03-29T11:08:47.910 INFO:teuthology.orchestra.run.smithi196.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 10 MB/s | 34 MB 00:03 2024-03-29T11:08:48.269 INFO:teuthology.orchestra.run.smithi196.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 8.0 MB/s | 28 MB 00:03 2024-03-29T11:08:48.272 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:08:48.272 INFO:teuthology.orchestra.run.smithi196.stdout:Total 21 MB/s | 76 MB 00:03 2024-03-29T11:08:48.272 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-29T11:08:48.638 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-29T11:08:48.638 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-29T11:08:49.492 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-29T11:08:49.493 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-29T11:08:49.879 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - AppStream 898 kB/s | 19 MB 00:21 2024-03-29T11:08:51.779 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-29T11:08:52.347 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:08:52.427 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:08:54.004 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-29T11:08:55.167 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:08:55.296 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:08:55.522 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - Extras packages 41 kB/s | 16 kB 00:00 2024-03-29T11:08:56.690 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 20 MB/s | 21 MB 00:01 2024-03-29T11:09:00.010 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:09:00.344 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:01.781 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-29T11:09:04.047 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-29T11:09:04.047 INFO:teuthology.orchestra.run.smithi086.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-29T11:09:04.144 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T11:09:04.149 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:09:04.150 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-29T11:09:04.150 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:09:04.150 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-29T11:09:04.150 INFO:teuthology.orchestra.run.smithi086.stdout: kernel x86_64 5.14.0-434.el9 CentOS-BaseOS 192 k 2024-03-29T11:09:04.150 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-29T11:09:04.150 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-core x86_64 5.14.0-434.el9 CentOS-BaseOS 15 M 2024-03-29T11:09:04.150 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules x86_64 5.14.0-434.el9 CentOS-BaseOS 34 M 2024-03-29T11:09:04.150 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-core x86_64 5.14.0-434.el9 CentOS-BaseOS 28 M 2024-03-29T11:09:04.150 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:09:04.150 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-29T11:09:04.150 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:09:04.150 INFO:teuthology.orchestra.run.smithi086.stdout:Install 4 Packages 2024-03-29T11:09:04.150 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:09:04.151 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 76 M 2024-03-29T11:09:04.151 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 123 M 2024-03-29T11:09:04.151 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-29T11:09:04.299 INFO:teuthology.orchestra.run.smithi086.stdout:(1/4): kernel-5.14.0-434.el9.x86_64.rpm 1.3 MB/s | 192 kB 00:00 2024-03-29T11:09:04.858 INFO:teuthology.orchestra.run.smithi086.stdout:(2/4): kernel-core-5.14.0-434.el9.x86_64.rpm 21 MB/s | 15 MB 00:00 2024-03-29T11:09:05.065 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:05.400 INFO:teuthology.orchestra.run.smithi086.stdout:(3/4): kernel-modules-5.14.0-434.el9.x86_64.rpm 27 MB/s | 34 MB 00:01 2024-03-29T11:09:05.642 INFO:teuthology.orchestra.run.smithi086.stdout:(4/4): kernel-modules-core-5.14.0-434.el9.x86_6 21 MB/s | 28 MB 00:01 2024-03-29T11:09:05.645 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:09:05.645 INFO:teuthology.orchestra.run.smithi086.stdout:Total 51 MB/s | 76 MB 00:01 2024-03-29T11:09:05.645 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-29T11:09:05.999 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-29T11:09:05.999 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-29T11:09:06.845 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-29T11:09:06.846 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-29T11:09:09.142 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-29T11:09:09.783 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-modules-core-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:09:09.867 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:09:12.605 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:09:12.743 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:09:17.464 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:09:17.575 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:22.317 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:34.338 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:34.338 INFO:teuthology.orchestra.run.smithi196.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:09:34.338 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:09:34.381 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:34.752 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:34.752 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:09:34.752 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:09:34.752 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:09:35.170 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:35.170 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:09:35.170 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-29T11:09:35.170 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:35.170 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:09:35.170 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:09:35.170 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:09:35.170 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:09:35.170 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T11:09:35.483 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:09:35.589 INFO:teuthology.orchestra.run.smithi196.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:35.589 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:09:35.590 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:09:35.590 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:09:35.590 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:09:35.590 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-03-29T11:09:35.590 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi196.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:09:35.590 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:09:35.670 INFO:teuthology.orchestra.run.smithi196.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:35.671 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:09:35.671 DEBUG:teuthology.orchestra.run.smithi196:> sudo rpm -qi grub2-tools 2024-03-29T11:09:35.706 INFO:teuthology.orchestra.run.smithi196.stdout:Name : grub2-tools 2024-03-29T11:09:35.706 INFO:teuthology.orchestra.run.smithi196.stdout:Epoch : 1 2024-03-29T11:09:35.706 INFO:teuthology.orchestra.run.smithi196.stdout:Version : 2.06 2024-03-29T11:09:35.707 INFO:teuthology.orchestra.run.smithi196.stdout:Release : 68.el9 2024-03-29T11:09:35.707 INFO:teuthology.orchestra.run.smithi196.stdout:Architecture: x86_64 2024-03-29T11:09:35.707 INFO:teuthology.orchestra.run.smithi196.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:09:35.707 INFO:teuthology.orchestra.run.smithi196.stdout:Group : Unspecified 2024-03-29T11:09:35.707 INFO:teuthology.orchestra.run.smithi196.stdout:Size : 8264254 2024-03-29T11:09:35.707 INFO:teuthology.orchestra.run.smithi196.stdout:License : GPLv3+ 2024-03-29T11:09:35.707 INFO:teuthology.orchestra.run.smithi196.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:09:35.707 INFO:teuthology.orchestra.run.smithi196.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:09:35.707 INFO:teuthology.orchestra.run.smithi196.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:09:35.707 INFO:teuthology.orchestra.run.smithi196.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:09:35.707 INFO:teuthology.orchestra.run.smithi196.stdout:Packager : builder@centos.org 2024-03-29T11:09:35.707 INFO:teuthology.orchestra.run.smithi196.stdout:Vendor : CentOS 2024-03-29T11:09:35.707 INFO:teuthology.orchestra.run.smithi196.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:09:35.707 INFO:teuthology.orchestra.run.smithi196.stdout:Summary : Support tools for GRUB. 2024-03-29T11:09:35.708 INFO:teuthology.orchestra.run.smithi196.stdout:Description : 2024-03-29T11:09:35.708 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:09:35.708 INFO:teuthology.orchestra.run.smithi196.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:09:35.708 INFO:teuthology.orchestra.run.smithi196.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:09:35.708 INFO:teuthology.orchestra.run.smithi196.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:09:35.708 INFO:teuthology.orchestra.run.smithi196.stdout:hardware devices. 2024-03-29T11:09:35.708 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:09:35.708 INFO:teuthology.orchestra.run.smithi196.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:09:35.709 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:09:35.709 INFO:teuthology.task.kernel:Updating grub on smithi196 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:09:35.709 DEBUG:teuthology.orchestra.run.smithi196:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:09:36.267 INFO:teuthology.orchestra.run.smithi196.stderr:Generating grub configuration file ... 2024-03-29T11:09:37.690 INFO:teuthology.orchestra.run.smithi196.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:09:37.726 INFO:teuthology.orchestra.run.smithi196.stderr:done 2024-03-29T11:09:37.728 DEBUG:teuthology.orchestra.run.smithi196:> mktemp 2024-03-29T11:09:37.743 INFO:teuthology.orchestra.run.smithi196.stdout:/tmp/tmp.kYvefU7FHe 2024-03-29T11:09:37.743 DEBUG:teuthology.orchestra.run.smithi196:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.kYvefU7FHe 2024-03-29T11:09:37.824 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0666 /tmp/tmp.kYvefU7FHe 2024-03-29T11:09:37.967 DEBUG:teuthology.orchestra.remote:smithi196:/tmp/tmp.kYvefU7FHe is 6KB 2024-03-29T11:09:38.016 DEBUG:teuthology.orchestra.run.smithi196:> rm -fr /tmp/tmp.kYvefU7FHe 2024-03-29T11:09:38.031 DEBUG:teuthology.orchestra.run.smithi196:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:09:38.105 INFO:teuthology.orchestra.run.smithi196.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:09:38.105 INFO:teuthology.orchestra.run.smithi196.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:09:38.105 INFO:teuthology.orchestra.run.smithi196.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:09:38.106 INFO:teuthology.orchestra.run.smithi196.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:09:38.107 DEBUG:teuthology.orchestra.run.smithi196:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:09:38.249 DEBUG:teuthology.orchestra.run.smithi196:> sudo shutdown -r now 2024-03-29T11:09:51.757 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:51.758 INFO:teuthology.orchestra.run.smithi086.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-29T11:09:51.758 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:09:51.800 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-modules-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:52.232 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: kernel-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:52.232 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-5.14.0-434.el9.x86_64 1/4 2024-03-29T11:09:52.232 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-core-5.14.0-434.el9.x86_64 2/4 2024-03-29T11:09:52.233 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-modules-5.14.0-434.el9.x86_64 3/4 2024-03-29T11:09:52.646 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : kernel-modules-core-5.14.0-434.el9.x86_64 4/4 2024-03-29T11:09:52.646 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:09:52.646 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-29T11:09:52.646 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:52.646 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-core-5.14.0-434.el9.x86_64 2024-03-29T11:09:52.646 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-5.14.0-434.el9.x86_64 2024-03-29T11:09:52.646 INFO:teuthology.orchestra.run.smithi086.stdout: kernel-modules-core-5.14.0-434.el9.x86_64 2024-03-29T11:09:52.646 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:09:52.646 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T11:09:52.982 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:09:53.121 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:53.121 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:09:53.121 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-434.el9.x86_64 2024-03-29T11:09:53.122 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-29T11:09:53.122 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-29T11:09:53.122 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-03-29T11:09:53.122 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi086.front.sepia.ceph.com, path=None, version=distro) 2024-03-29T11:09:53.122 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q kernel | sort -rV | head -n 1 2024-03-29T11:09:53.202 INFO:teuthology.orchestra.run.smithi086.stdout:kernel-5.14.0-434.el9.x86_64 2024-03-29T11:09:53.202 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-434.el9.x86_64 2024-03-29T11:09:53.202 DEBUG:teuthology.orchestra.run.smithi086:> sudo rpm -qi grub2-tools 2024-03-29T11:09:53.243 INFO:teuthology.orchestra.run.smithi086.stdout:Name : grub2-tools 2024-03-29T11:09:53.243 INFO:teuthology.orchestra.run.smithi086.stdout:Epoch : 1 2024-03-29T11:09:53.243 INFO:teuthology.orchestra.run.smithi086.stdout:Version : 2.06 2024-03-29T11:09:53.243 INFO:teuthology.orchestra.run.smithi086.stdout:Release : 68.el9 2024-03-29T11:09:53.243 INFO:teuthology.orchestra.run.smithi086.stdout:Architecture: x86_64 2024-03-29T11:09:53.243 INFO:teuthology.orchestra.run.smithi086.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-29T11:09:53.243 INFO:teuthology.orchestra.run.smithi086.stdout:Group : Unspecified 2024-03-29T11:09:53.243 INFO:teuthology.orchestra.run.smithi086.stdout:Size : 8264254 2024-03-29T11:09:53.243 INFO:teuthology.orchestra.run.smithi086.stdout:License : GPLv3+ 2024-03-29T11:09:53.243 INFO:teuthology.orchestra.run.smithi086.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-29T11:09:53.243 INFO:teuthology.orchestra.run.smithi086.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-29T11:09:53.243 INFO:teuthology.orchestra.run.smithi086.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-29T11:09:53.244 INFO:teuthology.orchestra.run.smithi086.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-29T11:09:53.244 INFO:teuthology.orchestra.run.smithi086.stdout:Packager : builder@centos.org 2024-03-29T11:09:53.244 INFO:teuthology.orchestra.run.smithi086.stdout:Vendor : CentOS 2024-03-29T11:09:53.244 INFO:teuthology.orchestra.run.smithi086.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-29T11:09:53.244 INFO:teuthology.orchestra.run.smithi086.stdout:Summary : Support tools for GRUB. 2024-03-29T11:09:53.244 INFO:teuthology.orchestra.run.smithi086.stdout:Description : 2024-03-29T11:09:53.244 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:09:53.244 INFO:teuthology.orchestra.run.smithi086.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-29T11:09:53.244 INFO:teuthology.orchestra.run.smithi086.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-29T11:09:53.244 INFO:teuthology.orchestra.run.smithi086.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-29T11:09:53.244 INFO:teuthology.orchestra.run.smithi086.stdout:hardware devices. 2024-03-29T11:09:53.244 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:09:53.244 INFO:teuthology.orchestra.run.smithi086.stdout:This subpackage provides tools for support of all platforms. 2024-03-29T11:09:53.245 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-29T11:09:53.245 INFO:teuthology.task.kernel:Updating grub on smithi086 to boot 5.14.0-434.el9.x86_64 2024-03-29T11:09:53.245 DEBUG:teuthology.orchestra.run.smithi086:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-29T11:09:53.841 INFO:teuthology.orchestra.run.smithi086.stderr:Generating grub configuration file ... 2024-03-29T11:09:55.219 INFO:teuthology.orchestra.run.smithi086.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-29T11:09:55.248 INFO:teuthology.orchestra.run.smithi086.stderr:done 2024-03-29T11:09:55.251 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2024-03-29T11:09:55.265 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.lyAtVNiaz3 2024-03-29T11:09:55.266 DEBUG:teuthology.orchestra.run.smithi086:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.lyAtVNiaz3 2024-03-29T11:09:55.335 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.lyAtVNiaz3 2024-03-29T11:09:55.474 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.lyAtVNiaz3 is 6KB 2024-03-29T11:09:55.522 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.lyAtVNiaz3 2024-03-29T11:09:55.537 DEBUG:teuthology.orchestra.run.smithi086:> sudo /bin/ls /boot/loader/entries || true 2024-03-29T11:09:55.601 INFO:teuthology.orchestra.run.smithi086.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-29T11:09:55.601 INFO:teuthology.orchestra.run.smithi086.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-29T11:09:55.601 INFO:teuthology.orchestra.run.smithi086.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-29T11:09:55.601 INFO:teuthology.orchestra.run.smithi086.stdout:15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64.conf 2024-03-29T11:09:55.602 DEBUG:teuthology.orchestra.run.smithi086:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-434.el9.x86_64 2024-03-29T11:09:55.734 DEBUG:teuthology.orchestra.run.smithi086:> sudo shutdown -r now 2024-03-29T11:10:08.256 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-03-29T11:10:08.257 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-03-29T11:10:08.258 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:25.754 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-03-29T11:10:25.754 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-03-29T11:10:25.755 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:26.590 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.196 2024-03-29T11:10:35.598 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-03-29T11:10:35.599 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:44.254 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-03-29T11:10:53.262 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-03-29T11:10:53.263 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:10:56.318 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.86 2024-03-29T11:11:08.330 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-03-29T11:11:08.331 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:11:35.626 DEBUG:teuthology.orchestra.remote:timed out 2024-03-29T11:11:40.301 DEBUG:teuthology.orchestra.run.smithi086:> true 2024-03-29T11:11:40.744 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-03-29T11:11:40.745 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:11:40.745 DEBUG:teuthology.orchestra.run.smithi086:> uname -r 2024-03-29T11:11:40.760 INFO:teuthology.orchestra.run.smithi086.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:11:40.761 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:11:40.761 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:11:40.761 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:11:41.762 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-03-29T11:11:41.762 DEBUG:teuthology.orchestra.run.smithi086:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:11:41.846 INFO:teuthology.orchestra.run.smithi086.stdout:ttyS1 2024-03-29T11:11:41.884 DEBUG:teuthology.parallel:result is None 2024-03-29T11:11:47.628 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-03-29T11:11:47.629 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:11:47.994 DEBUG:teuthology.orchestra.run.smithi196:> true 2024-03-29T11:11:48.358 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-03-29T11:11:48.358 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-434.el9.x86_64"... 2024-03-29T11:11:48.358 DEBUG:teuthology.orchestra.run.smithi196:> uname -r 2024-03-29T11:11:48.374 INFO:teuthology.orchestra.run.smithi196.stdout:5.14.0-434.el9.x86_64 2024-03-29T11:11:48.374 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-434.el9.x86_64 vs 5.14.0-434.el9.x86_64 2024-03-29T11:11:48.375 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-29T11:11:48.375 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-29T11:11:49.375 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-03-29T11:11:49.375 DEBUG:teuthology.orchestra.run.smithi196:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-29T11:11:49.463 INFO:teuthology.orchestra.run.smithi196.stdout:ttyS1 2024-03-29T11:11:49.499 DEBUG:teuthology.parallel:result is None 2024-03-29T11:11:49.500 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-29T11:11:49.507 INFO:teuthology.task.internal:Creating test directory... 2024-03-29T11:11:49.508 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:11:49.511 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-29T11:11:49.529 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-29T11:11:49.561 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-29T11:11:49.593 INFO:teuthology.task.internal:Creating archive directory... 2024-03-29T11:11:49.594 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:11:49.596 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-29T11:11:49.888 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-29T11:11:49.895 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-29T11:11:49.895 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:11:49.898 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-29T11:11:49.929 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:11:49.934 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:11:49.943 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:11:49.949 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-29T11:11:49.950 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-29T11:11:49.958 INFO:teuthology.task.internal:Configuring sudo... 2024-03-29T11:11:49.958 DEBUG:teuthology.orchestra.run.smithi086:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:11:49.987 DEBUG:teuthology.orchestra.run.smithi196:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-29T11:11:50.021 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-29T11:11:50.032 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-29T11:11:50.032 DEBUG:teuthology.orchestra.run.smithi086:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:11:50.059 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-29T11:11:50.077 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:11:50.159 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:11:50.247 DEBUG:teuthology.orchestra.run.smithi086:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:11:50.285 DEBUG:teuthology.orchestra.run.smithi086:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:11:50.334 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T11:11:50.334 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:11:50.401 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:11:50.443 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-29T11:11:50.494 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:11:50.577 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-29T11:11:50.623 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T11:11:50.623 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-29T11:11:50.691 DEBUG:teuthology.orchestra.run.smithi086:> sudo service rsyslog restart 2024-03-29T11:11:50.694 DEBUG:teuthology.orchestra.run.smithi196:> sudo service rsyslog restart 2024-03-29T11:11:50.753 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:11:50.776 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:11:51.126 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-29T11:11:51.134 INFO:teuthology.task.internal:Starting timer... 2024-03-29T11:11:51.134 INFO:teuthology.run_tasks:Running task pcp... 2024-03-29T11:11:51.165 INFO:teuthology.run_tasks:Running task selinux... 2024-03-29T11:11:51.204 DEBUG:teuthology.orchestra.run.smithi086:> sudo service auditd rotate 2024-03-29T11:11:51.256 INFO:teuthology.orchestra.run.smithi086.stdout:Rotating logs: 2024-03-29T11:11:51.258 DEBUG:teuthology.orchestra.run.smithi196:> sudo service auditd rotate 2024-03-29T11:11:51.320 INFO:teuthology.orchestra.run.smithi196.stdout:Rotating logs: 2024-03-29T11:11:51.321 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-29T11:11:51.322 DEBUG:teuthology.orchestra.run.smithi086:> /usr/sbin/getenforce 2024-03-29T11:11:51.347 INFO:teuthology.orchestra.run.smithi086.stdout:Permissive 2024-03-29T11:11:51.347 DEBUG:teuthology.orchestra.run.smithi196:> /usr/sbin/getenforce 2024-03-29T11:11:51.370 INFO:teuthology.orchestra.run.smithi196.stdout:Permissive 2024-03-29T11:11:51.371 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi086.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi196.front.sepia.ceph.com': 'permissive'} 2024-03-29T11:11:51.371 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:11:51.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:11:51.417 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:11:51.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:11:51.446 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-29T11:11:51.446 DEBUG:teuthology.orchestra.run.smithi086:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:11:51.490 DEBUG:teuthology.orchestra.run.smithi196:> sudo /usr/sbin/setenforce permissive 2024-03-29T11:11:51.522 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-29T11:11:51.532 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-29T11:11:51.549 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-29T11:11:51.550 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi086.front.sepia.ceph.com,smithi196.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-29T11:16:13.797 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi086.front.sepia.ceph.com'), Remote(name='ubuntu@smithi196.front.sepia.ceph.com')] 2024-03-29T11:16:13.798 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-03-29T11:16:13.799 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi086.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:16:13.886 DEBUG:teuthology.orchestra.run.smithi086:> true 2024-03-29T11:16:13.961 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi086.front.sepia.ceph.com' 2024-03-29T11:16:13.961 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-03-29T11:16:13.962 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-29T11:16:14.046 DEBUG:teuthology.orchestra.run.smithi196:> true 2024-03-29T11:16:14.120 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-03-29T11:16:14.121 INFO:teuthology.run_tasks:Running task clock... 2024-03-29T11:16:14.133 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-29T11:16:14.133 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:16:14.133 DEBUG:teuthology.orchestra.run.smithi086:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:16:14.136 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-29T11:16:14.136 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:16:14.164 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:16:14.180 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:16:14.191 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-29T11:16:14.207 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-29T11:16:14.209 INFO:teuthology.orchestra.run.smithi086.stderr:sudo: ntpd: command not found 2024-03-29T11:16:14.223 INFO:teuthology.orchestra.run.smithi086.stdout:506 Cannot talk to daemon 2024-03-29T11:16:14.236 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: ntpd: command not found 2024-03-29T11:16:14.239 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:16:14.250 INFO:teuthology.orchestra.run.smithi196.stdout:506 Cannot talk to daemon 2024-03-29T11:16:14.255 INFO:teuthology.orchestra.run.smithi086.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:16:14.266 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-29T11:16:14.283 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-29T11:16:14.307 INFO:teuthology.orchestra.run.smithi086.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:16:14.310 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:16:14.310 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2024-03-29T11:16:14.310 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:16:14.310 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:16:14.311 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:16:14.311 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:16:14.336 INFO:teuthology.orchestra.run.smithi196.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:16:14.339 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:16:14.339 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== 2024-03-29T11:16:14.339 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:16:14.339 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:16:14.339 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:16:14.339 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:16:14.340 INFO:teuthology.run_tasks:Running task install... 2024-03-29T11:16:14.348 DEBUG:teuthology.task.install:project ceph 2024-03-29T11:16:14.348 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}} 2024-03-29T11:16:14.348 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'} 2024-03-29T11:16:14.348 INFO:teuthology.task.install:Using flavor: default 2024-03-29T11:16:14.354 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-29T11:16:14.354 INFO:teuthology.task.install:extra packages: [] 2024-03-29T11:16:14.355 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:16:14.355 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:16:14.357 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'tag': None, 'wait_for_package': False} 2024-03-29T11:16:14.357 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:16:14.510 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:16:14.510 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:16:14.621 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:16:14.621 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T11:16:14.621 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:16:14.648 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:16:14.649 DEBUG:teuthology.orchestra.run.smithi086:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:16:14.717 DEBUG:teuthology.orchestra.run.smithi086:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:16:14.732 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/ 2024-03-29T11:16:14.732 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1439.g5dcb190d 2024-03-29T11:16:14.803 DEBUG:teuthology.orchestra.run.smithi086:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:16:14.841 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/wip-amat-fix-encode-payload/5dcb190da270440ef141d088eb27a9fdb8afc5f2/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-29T11:16:14.841 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T11:16:14.841 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-29T11:16:14.870 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-29T11:16:14.870 DEBUG:teuthology.orchestra.run.smithi196:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5dcb190da270440ef141d088eb27a9fdb8afc5f2/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-29T11:16:14.871 INFO:teuthology.orchestra.run.smithi086.stdout:check_obsoletes = 1 2024-03-29T11:16:14.871 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum clean all 2024-03-29T11:16:14.939 DEBUG:teuthology.orchestra.run.smithi196:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-29T11:16:15.019 DEBUG:teuthology.orchestra.run.smithi196:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-29T11:16:15.085 INFO:teuthology.orchestra.run.smithi196.stdout:check_obsoletes = 1 2024-03-29T11:16:15.087 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum clean all 2024-03-29T11:16:15.115 INFO:teuthology.orchestra.run.smithi086.stdout:74 files removed 2024-03-29T11:16:15.142 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-radosgw 2024-03-29T11:16:15.338 INFO:teuthology.orchestra.run.smithi196.stdout:74 files removed 2024-03-29T11:16:15.365 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-radosgw 2024-03-29T11:16:16.137 INFO:teuthology.orchestra.run.smithi086.stdout:ceph packages for x86_64 118 kB/s | 82 kB 00:00 2024-03-29T11:16:16.368 INFO:teuthology.orchestra.run.smithi196.stdout:ceph packages for x86_64 117 kB/s | 82 kB 00:00 2024-03-29T11:16:16.721 INFO:teuthology.orchestra.run.smithi086.stdout:ceph noarch packages 19 kB/s | 10 kB 00:00 2024-03-29T11:16:16.834 INFO:teuthology.orchestra.run.smithi196.stdout:ceph noarch packages 24 kB/s | 10 kB 00:00 2024-03-29T11:16:17.335 INFO:teuthology.orchestra.run.smithi196.stdout:ceph source packages 4.1 kB/s | 1.9 kB 00:00 2024-03-29T11:16:17.464 INFO:teuthology.orchestra.run.smithi086.stdout:ceph source packages 2.7 kB/s | 1.9 kB 00:00 2024-03-29T11:16:17.912 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-29T11:16:18.307 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS-9 - AppStream 8.3 MB/s | 6.7 MB 00:00 2024-03-29T11:16:19.521 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS-9 - BaseOS 4.0 MB/s | 2.0 MB 00:00 2024-03-29T11:16:19.708 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS-9 - BaseOS 6.8 MB/s | 2.0 MB 00:00 2024-03-29T11:16:20.457 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - BaseOS 19 MB/s | 8.0 MB 00:00 2024-03-29T11:16:20.503 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-29T11:16:22.391 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-29T11:16:23.187 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-03-29T11:16:28.146 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - CRB 4.3 MB/s | 6.1 MB 00:01 2024-03-29T11:16:30.050 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - Extras packages 28 kB/s | 16 kB 00:00 2024-03-29T11:16:33.652 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - CRB 808 kB/s | 6.1 MB 00:07 2024-03-29T11:16:34.673 INFO:teuthology.orchestra.run.smithi196.stdout:Extra Packages for Enterprise Linux 4.6 MB/s | 21 MB 00:04 2024-03-29T11:16:35.527 INFO:teuthology.orchestra.run.smithi086.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-03-29T11:16:36.748 INFO:teuthology.orchestra.run.smithi086.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-03-29T11:16:39.673 INFO:teuthology.orchestra.run.smithi196.stdout:lab-extras 12 kB/s | 1.7 kB 00:00 2024-03-29T11:16:41.796 INFO:teuthology.orchestra.run.smithi086.stdout:lab-extras 13 kB/s | 1.7 kB 00:00 2024-03-29T11:16:42.294 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T11:16:42.311 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:16:42.311 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-29T11:16:42.311 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:16:42.311 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-29T11:16:42.311 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:16:42.311 INFO:teuthology.orchestra.run.smithi196.stdout:Upgrading: 2024-03-29T11:16:42.312 INFO:teuthology.orchestra.run.smithi196.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:16:42.312 INFO:teuthology.orchestra.run.smithi196.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:16:42.312 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-29T11:16:42.312 INFO:teuthology.orchestra.run.smithi196.stdout: boost-program-options 2024-03-29T11:16:42.312 INFO:teuthology.orchestra.run.smithi196.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:16:42.312 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:16:42.312 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:16:42.312 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:16:42.312 INFO:teuthology.orchestra.run.smithi196.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:16:42.312 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:16:42.312 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:16:42.312 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:16:42.313 INFO:teuthology.orchestra.run.smithi196.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:16:42.313 INFO:teuthology.orchestra.run.smithi196.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:16:42.313 INFO:teuthology.orchestra.run.smithi196.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:16:42.313 INFO:teuthology.orchestra.run.smithi196.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:16:42.313 INFO:teuthology.orchestra.run.smithi196.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:16:42.313 INFO:teuthology.orchestra.run.smithi196.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:16:42.313 INFO:teuthology.orchestra.run.smithi196.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:16:42.313 INFO:teuthology.orchestra.run.smithi196.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:16:42.313 INFO:teuthology.orchestra.run.smithi196.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:16:42.313 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-argparse 2024-03-29T11:16:42.313 INFO:teuthology.orchestra.run.smithi196.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:16:42.313 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:16:42.313 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:16:42.314 INFO:teuthology.orchestra.run.smithi196.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:16:42.314 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:16:42.314 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:16:42.314 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:16:42.314 INFO:teuthology.orchestra.run.smithi196.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:16:42.314 INFO:teuthology.orchestra.run.smithi196.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:16:42.314 INFO:teuthology.orchestra.run.smithi196.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:16:42.314 INFO:teuthology.orchestra.run.smithi196.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:16:42.314 INFO:teuthology.orchestra.run.smithi196.stdout:Installing weak dependencies: 2024-03-29T11:16:42.314 INFO:teuthology.orchestra.run.smithi196.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:16:42.314 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:16:42.315 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-29T11:16:42.316 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:16:42.316 INFO:teuthology.orchestra.run.smithi196.stdout:Install 30 Packages 2024-03-29T11:16:42.316 INFO:teuthology.orchestra.run.smithi196.stdout:Upgrade 2 Packages 2024-03-29T11:16:42.316 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:16:42.316 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 58 M 2024-03-29T11:16:42.316 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-29T11:16:44.407 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T11:16:44.414 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:16:44.415 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-29T11:16:44.415 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:16:44.415 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-29T11:16:44.415 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 9.7 M 2024-03-29T11:16:44.415 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrading: 2024-03-29T11:16:44.415 INFO:teuthology.orchestra.run.smithi086.stdout: librados2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.4 M 2024-03-29T11:16:44.415 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:16:44.416 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-29T11:16:44.416 INFO:teuthology.orchestra.run.smithi086.stdout: boost-program-options 2024-03-29T11:16:44.416 INFO:teuthology.orchestra.run.smithi086.stdout: x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-29T11:16:44.416 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.3 M 2024-03-29T11:16:44.416 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 20 M 2024-03-29T11:16:44.416 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 25 k 2024-03-29T11:16:44.417 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-29T11:16:44.417 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-29T11:16:44.417 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-29T11:16:44.417 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 742 k 2024-03-29T11:16:44.417 INFO:teuthology.orchestra.run.smithi086.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-29T11:16:44.417 INFO:teuthology.orchestra.run.smithi086.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-29T11:16:44.417 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-29T11:16:44.417 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 498 k 2024-03-29T11:16:44.417 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-29T11:16:44.417 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2 x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 5.1 M 2024-03-29T11:16:44.418 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-29T11:16:44.418 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-29T11:16:44.418 INFO:teuthology.orchestra.run.smithi086.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-29T11:16:44.418 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse 2024-03-29T11:16:44.418 INFO:teuthology.orchestra.run.smithi086.stdout: x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 45 k 2024-03-29T11:16:44.418 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 131 k 2024-03-29T11:16:44.418 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 162 k 2024-03-29T11:16:44.418 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-29T11:16:44.418 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 320 k 2024-03-29T11:16:44.418 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 298 k 2024-03-29T11:16:44.418 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 99 k 2024-03-29T11:16:44.418 INFO:teuthology.orchestra.run.smithi086.stdout: qatlib x86_64 24.02.0-1.el9 CentOS-AppStream 220 k 2024-03-29T11:16:44.418 INFO:teuthology.orchestra.run.smithi086.stdout: qatzip-libs x86_64 1.2.0-1.el9 CentOS-AppStream 46 k 2024-03-29T11:16:44.418 INFO:teuthology.orchestra.run.smithi086.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-29T11:16:44.419 INFO:teuthology.orchestra.run.smithi086.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-29T11:16:44.419 INFO:teuthology.orchestra.run.smithi086.stdout:Installing weak dependencies: 2024-03-29T11:16:44.419 INFO:teuthology.orchestra.run.smithi086.stdout: qatlib-service x86_64 24.02.0-1.el9 CentOS-AppStream 35 k 2024-03-29T11:16:44.419 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:16:44.419 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-29T11:16:44.419 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:16:44.419 INFO:teuthology.orchestra.run.smithi086.stdout:Install 30 Packages 2024-03-29T11:16:44.419 INFO:teuthology.orchestra.run.smithi086.stdout:Upgrade 2 Packages 2024-03-29T11:16:44.419 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:16:44.420 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 58 M 2024-03-29T11:16:44.420 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-29T11:16:47.031 INFO:teuthology.orchestra.run.smithi196.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 1.1 MB/s | 5.3 MB 00:04 2024-03-29T11:16:47.148 INFO:teuthology.orchestra.run.smithi196.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 216 kB/s | 25 kB 00:00 2024-03-29T11:16:48.132 INFO:teuthology.orchestra.run.smithi196.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 754 kB/s | 742 kB 00:00 2024-03-29T11:16:48.981 INFO:teuthology.orchestra.run.smithi196.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 587 kB/s | 498 kB 00:00 2024-03-29T11:16:49.580 INFO:teuthology.orchestra.run.smithi086.stdout:(1/32): ceph-base-19.0.0-1439.g5dcb190d.el9.x86 1.1 MB/s | 5.3 MB 00:04 2024-03-29T11:16:49.765 INFO:teuthology.orchestra.run.smithi086.stdout:(2/32): ceph-selinux-19.0.0-1439.g5dcb190d.el9. 275 kB/s | 25 kB 00:00 2024-03-29T11:16:50.397 INFO:teuthology.orchestra.run.smithi086.stdout:(3/32): libcephfs2-19.0.0-1439.g5dcb190d.el9.x8 1.0 MB/s | 742 kB 00:00 2024-03-29T11:16:50.964 INFO:teuthology.orchestra.run.smithi086.stdout:(4/32): libradosstriper1-19.0.0-1439.g5dcb190d. 879 kB/s | 498 kB 00:00 2024-03-29T11:16:53.414 INFO:teuthology.orchestra.run.smithi086.stdout:(5/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9. 1.1 MB/s | 9.7 MB 00:08 2024-03-29T11:16:53.514 INFO:teuthology.orchestra.run.smithi086.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 453 kB/s | 45 kB 00:00 2024-03-29T11:16:53.764 INFO:teuthology.orchestra.run.smithi086.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 525 kB/s | 131 kB 00:00 2024-03-29T11:16:53.998 INFO:teuthology.orchestra.run.smithi086.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 694 kB/s | 162 kB 00:00 2024-03-29T11:16:54.490 INFO:teuthology.orchestra.run.smithi086.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 652 kB/s | 320 kB 00:00 2024-03-29T11:16:54.973 INFO:teuthology.orchestra.run.smithi086.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 616 kB/s | 298 kB 00:00 2024-03-29T11:16:55.232 INFO:teuthology.orchestra.run.smithi086.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 384 kB/s | 99 kB 00:00 2024-03-29T11:16:55.366 INFO:teuthology.orchestra.run.smithi086.stdout:(12/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_ 1.2 MB/s | 5.1 MB 00:04 2024-03-29T11:16:55.391 INFO:teuthology.orchestra.run.smithi086.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 667 kB/s | 106 kB 00:00 2024-03-29T11:16:55.416 INFO:teuthology.orchestra.run.smithi086.stdout:(14/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-29T11:16:55.441 INFO:teuthology.orchestra.run.smithi086.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 888 kB/s | 44 kB 00:00 2024-03-29T11:16:55.475 INFO:teuthology.orchestra.run.smithi086.stdout:(16/32): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-29T11:16:55.500 INFO:teuthology.orchestra.run.smithi086.stdout:(17/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.8 MB/s | 282 kB 00:00 2024-03-29T11:16:55.526 INFO:teuthology.orchestra.run.smithi086.stdout:(18/32): python3-prettytable-0.7.2-27.el9.noarc 831 kB/s | 41 kB 00:00 2024-03-29T11:16:55.551 INFO:teuthology.orchestra.run.smithi086.stdout:(19/32): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 220 kB 00:00 2024-03-29T11:16:55.576 INFO:teuthology.orchestra.run.smithi086.stdout:(20/32): qatlib-service-24.02.0-1.el9.x86_64.rp 698 kB/s | 35 kB 00:00 2024-03-29T11:16:55.601 INFO:teuthology.orchestra.run.smithi086.stdout:(21/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 925 kB/s | 46 kB 00:00 2024-03-29T11:16:55.869 INFO:teuthology.orchestra.run.smithi086.stdout:(22/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.0 MB/s | 309 kB 00:00 2024-03-29T11:16:55.961 INFO:teuthology.orchestra.run.smithi086.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 283 kB/s | 26 kB 00:00 2024-03-29T11:16:56.019 INFO:teuthology.orchestra.run.smithi086.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 833 kB/s | 49 kB 00:00 2024-03-29T11:16:56.078 INFO:teuthology.orchestra.run.smithi086.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.1 MB/s | 67 kB 00:00 2024-03-29T11:16:56.195 INFO:teuthology.orchestra.run.smithi086.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.0 MB/s | 838 kB 00:00 2024-03-29T11:16:56.261 INFO:teuthology.orchestra.run.smithi086.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-29T11:16:56.404 INFO:teuthology.orchestra.run.smithi086.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-29T11:16:59.537 INFO:teuthology.orchestra.run.smithi086.stdout:(29/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 1.1 MB/s | 3.4 MB 00:03 2024-03-29T11:16:59.830 INFO:teuthology.orchestra.run.smithi196.stdout:(5/32): librgw2-19.0.0-1439.g5dcb190d.el9.x86_6 481 kB/s | 5.1 MB 00:10 2024-03-29T11:16:59.947 INFO:teuthology.orchestra.run.smithi196.stdout:(6/32): python3-ceph-argparse-19.0.0-1439.g5dcb 388 kB/s | 45 kB 00:00 2024-03-29T11:17:00.222 INFO:teuthology.orchestra.run.smithi196.stdout:(7/32): python3-ceph-common-19.0.0-1439.g5dcb19 478 kB/s | 131 kB 00:00 2024-03-29T11:17:00.523 INFO:teuthology.orchestra.run.smithi196.stdout:(8/32): python3-cephfs-19.0.0-1439.g5dcb190d.el 540 kB/s | 162 kB 00:00 2024-03-29T11:17:01.114 INFO:teuthology.orchestra.run.smithi196.stdout:(9/32): python3-rados-19.0.0-1439.g5dcb190d.el9 542 kB/s | 320 kB 00:00 2024-03-29T11:17:01.598 INFO:teuthology.orchestra.run.smithi196.stdout:(10/32): python3-rbd-19.0.0-1439.g5dcb190d.el9. 616 kB/s | 298 kB 00:00 2024-03-29T11:17:01.799 INFO:teuthology.orchestra.run.smithi196.stdout:(11/32): python3-rgw-19.0.0-1439.g5dcb190d.el9. 496 kB/s | 99 kB 00:00 2024-03-29T11:17:01.907 INFO:teuthology.orchestra.run.smithi196.stdout:(12/32): boost-program-options-1.75.0-8.el9.x86 979 kB/s | 106 kB 00:00 2024-03-29T11:17:01.949 INFO:teuthology.orchestra.run.smithi196.stdout:(13/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.7 MB/s | 159 kB 00:00 2024-03-29T11:17:01.987 INFO:teuthology.orchestra.run.smithi086.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 1.3 MB/s | 3.1 MB 00:02 2024-03-29T11:17:01.991 INFO:teuthology.orchestra.run.smithi196.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.0 MB/s | 44 kB 00:00 2024-03-29T11:17:02.043 INFO:teuthology.orchestra.run.smithi196.stdout:(15/32): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-29T11:17:02.084 INFO:teuthology.orchestra.run.smithi196.stdout:(16/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-29T11:17:02.117 INFO:teuthology.orchestra.run.smithi196.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.2 MB/s | 41 kB 00:00 2024-03-29T11:17:02.151 INFO:teuthology.orchestra.run.smithi196.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 6.4 MB/s | 220 kB 00:00 2024-03-29T11:17:02.184 INFO:teuthology.orchestra.run.smithi196.stdout:(19/32): qatlib-service-24.02.0-1.el9.x86_64.rp 1.0 MB/s | 35 kB 00:00 2024-03-29T11:17:02.218 INFO:teuthology.orchestra.run.smithi196.stdout:(20/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 46 kB 00:00 2024-03-29T11:17:02.585 INFO:teuthology.orchestra.run.smithi196.stdout:(21/32): gperftools-libs-2.9.1-2.el9.x86_64.rpm 843 kB/s | 309 kB 00:00 2024-03-29T11:17:02.877 INFO:teuthology.orchestra.run.smithi196.stdout:(22/32): libarrow-9.0.0-10.el9.x86_64.rpm 15 MB/s | 4.4 MB 00:00 2024-03-29T11:17:02.944 INFO:teuthology.orchestra.run.smithi196.stdout:(23/32): libarrow-doc-9.0.0-10.el9.noarch.rpm 391 kB/s | 26 kB 00:00 2024-03-29T11:17:03.010 INFO:teuthology.orchestra.run.smithi196.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 730 kB/s | 49 kB 00:00 2024-03-29T11:17:03.078 INFO:teuthology.orchestra.run.smithi196.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-03-29T11:17:03.178 INFO:teuthology.orchestra.run.smithi196.stdout:(26/32): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.2 MB/s | 838 kB 00:00 2024-03-29T11:17:03.245 INFO:teuthology.orchestra.run.smithi196.stdout:(27/32): re2-20211101-3.el9.x86_64.rpm 2.8 MB/s | 192 kB 00:00 2024-03-29T11:17:03.353 INFO:teuthology.orchestra.run.smithi196.stdout:(28/32): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-03-29T11:17:03.637 INFO:teuthology.orchestra.run.smithi196.stdout:(29/32): ceph-radosgw-19.0.0-1439.g5dcb190d.el9 469 kB/s | 9.7 MB 00:21 2024-03-29T11:17:06.545 INFO:teuthology.orchestra.run.smithi196.stdout:(30/32): librbd1-19.0.0-1439.g5dcb190d.el9.x86_ 1.1 MB/s | 3.1 MB 00:02 2024-03-29T11:17:07.037 INFO:teuthology.orchestra.run.smithi196.stdout:(31/32): librados2-19.0.0-1439.g5dcb190d.el9.x8 938 kB/s | 3.4 MB 00:03 2024-03-29T11:17:09.670 INFO:teuthology.orchestra.run.smithi196.stdout:(32/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 746 kB/s | 20 MB 00:27 2024-03-29T11:17:09.674 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:09.675 INFO:teuthology.orchestra.run.smithi196.stdout:Total 2.1 MB/s | 58 MB 00:27 2024-03-29T11:17:09.675 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-29T11:17:09.713 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-29T11:17:09.713 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-29T11:17:09.970 INFO:teuthology.orchestra.run.smithi086.stdout:(31/32): ceph-common-19.0.0-1439.g5dcb190d.el9. 808 kB/s | 20 MB 00:25 2024-03-29T11:17:10.181 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-29T11:17:10.182 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-29T11:17:11.631 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-29T11:17:11.816 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:17:11.950 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:17:11.975 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:17:12.006 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:17:12.024 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:17:12.166 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:17:12.220 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:17:12.546 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:17:12.643 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:17:12.718 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:17:12.805 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:17:12.828 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:17:12.890 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:17:12.919 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:17:12.963 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:17:13.030 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:17:13.064 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:17:13.110 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:17:13.145 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:17:13.252 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:17:13.288 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:17:13.328 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:17:13.361 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:17:13.390 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:17:13.423 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:17:13.734 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:17:13.761 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:17:13.850 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:17:13.882 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:17:13.922 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:17:14.373 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:17:14.405 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:17:14.468 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:17:14.536 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:17:14.565 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:17:16.751 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:17:16.796 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:17:17.339 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:17:17.382 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:17:17.476 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:17:17.476 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:17:17.476 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:17:17.476 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:17:17.516 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:17:26.611 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:17:26.611 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /sys 2024-03-29T11:17:26.612 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /proc 2024-03-29T11:17:26.612 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /mnt 2024-03-29T11:17:26.612 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /var/tmp 2024-03-29T11:17:26.612 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /home 2024-03-29T11:17:26.612 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /root 2024-03-29T11:17:26.612 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /tmp 2024-03-29T11:17:26.612 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:17:27.527 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:17:27.550 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:17:27.550 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:27.551 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:17:27.551 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:17:27.551 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:17:27.551 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:17:27.552 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:17:27.629 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:17:27.629 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:17:27.919 INFO:teuthology.orchestra.run.smithi086.stdout:(32/32): libarrow-9.0.0-10.el9.x86_64.rpm 140 kB/s | 4.4 MB 00:32 2024-03-29T11:17:27.923 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:27.923 INFO:teuthology.orchestra.run.smithi086.stdout:Total 1.3 MB/s | 58 MB 00:43 2024-03-29T11:17:27.923 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-29T11:17:27.960 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-29T11:17:27.960 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-29T11:17:28.485 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-29T11:17:28.485 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-29T11:17:28.737 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:17:28.737 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:17:28.737 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:17:28.737 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:17:28.737 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:17:28.738 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:17:28.738 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:17:28.738 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:17:28.738 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:17:28.738 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:17:28.738 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:17:28.738 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:17:28.738 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:17:28.738 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:17:28.738 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:17:28.739 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:17:28.739 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:17:28.739 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:17:28.739 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:17:28.739 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:17:28.739 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:17:28.739 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:17:28.739 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:17:28.739 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:17:28.739 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:17:28.740 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:17:28.740 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:17:28.740 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:17:28.740 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:17:28.740 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:17:28.740 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:17:28.741 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:17:28.741 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:17:28.741 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:17:29.782 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:17:29.782 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:17:29.782 INFO:teuthology.orchestra.run.smithi196.stdout:Upgraded: 2024-03-29T11:17:29.782 INFO:teuthology.orchestra.run.smithi196.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:29.782 INFO:teuthology.orchestra.run.smithi196.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:29.782 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-29T11:17:29.782 INFO:teuthology.orchestra.run.smithi196.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:17:29.782 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:29.782 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:29.783 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:29.783 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:29.783 INFO:teuthology.orchestra.run.smithi196.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:17:29.783 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:17:29.783 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:17:29.783 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:29.783 INFO:teuthology.orchestra.run.smithi196.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:17:29.783 INFO:teuthology.orchestra.run.smithi196.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:17:29.783 INFO:teuthology.orchestra.run.smithi196.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:17:29.783 INFO:teuthology.orchestra.run.smithi196.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:29.783 INFO:teuthology.orchestra.run.smithi196.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:17:29.783 INFO:teuthology.orchestra.run.smithi196.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:29.783 INFO:teuthology.orchestra.run.smithi196.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:17:29.783 INFO:teuthology.orchestra.run.smithi196.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:17:29.783 INFO:teuthology.orchestra.run.smithi196.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:17:29.784 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:29.784 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:29.784 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:29.784 INFO:teuthology.orchestra.run.smithi196.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:17:29.784 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:29.784 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:29.784 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:29.784 INFO:teuthology.orchestra.run.smithi196.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:17:29.784 INFO:teuthology.orchestra.run.smithi196.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:17:29.784 INFO:teuthology.orchestra.run.smithi196.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:17:29.784 INFO:teuthology.orchestra.run.smithi196.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:17:29.784 INFO:teuthology.orchestra.run.smithi196.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:17:29.784 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:17:29.784 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T11:17:29.940 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-29T11:17:30.023 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-03-29T11:17:30.065 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-03-29T11:17:30.083 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-test 2024-03-29T11:17:30.090 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-03-29T11:17:30.114 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:17:30.129 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-03-29T11:17:30.271 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:17:30.326 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-03-29T11:17:30.639 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:00:51 ago on Fri 29 Mar 2024 11:16:39 AM UTC. 2024-03-29T11:17:30.689 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-03-29T11:17:30.725 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:17:30.792 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T11:17:30.792 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:17:30.792 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-29T11:17:30.792 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:17:30.793 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-29T11:17:30.793 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:17:30.793 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-29T11:17:30.793 INFO:teuthology.orchestra.run.smithi196.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:17:30.793 INFO:teuthology.orchestra.run.smithi196.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:17:30.793 INFO:teuthology.orchestra.run.smithi196.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:17:30.793 INFO:teuthology.orchestra.run.smithi196.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:17:30.793 INFO:teuthology.orchestra.run.smithi196.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:17:30.793 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:17:30.793 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-29T11:17:30.793 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:17:30.793 INFO:teuthology.orchestra.run.smithi196.stdout:Install 6 Packages 2024-03-29T11:17:30.793 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:17:30.794 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 48 M 2024-03-29T11:17:30.794 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 204 M 2024-03-29T11:17:30.794 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-29T11:17:30.816 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:17:30.904 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 8/34 2024-03-29T11:17:30.938 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:17:30.997 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 9/34 2024-03-29T11:17:31.042 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-03-29T11:17:31.086 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 11/34 2024-03-29T11:17:31.154 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:17:31.179 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:17:31.242 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 13/34 2024-03-29T11:17:31.261 INFO:teuthology.orchestra.run.smithi196.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-29T11:17:31.277 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-03-29T11:17:31.312 INFO:teuthology.orchestra.run.smithi196.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.8 MB/s | 300 kB 00:00 2024-03-29T11:17:31.346 INFO:teuthology.orchestra.run.smithi196.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 63 kB 00:00 2024-03-29T11:17:31.383 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : re2-1:20211101-3.el9.x86_64 15/34 2024-03-29T11:17:31.420 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 16/34 2024-03-29T11:17:31.460 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-03-29T11:17:31.493 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 18/34 2024-03-29T11:17:31.522 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 19/34 2024-03-29T11:17:31.554 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-03-29T11:17:31.862 INFO:teuthology.orchestra.run.smithi196.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 213 kB/s | 163 kB 00:00 2024-03-29T11:17:31.870 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-03-29T11:17:31.901 INFO:teuthology.orchestra.run.smithi086.stdout: Upgrading : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:17:31.981 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 22/34 2024-03-29T11:17:32.014 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 23/34 2024-03-29T11:17:32.057 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-03-29T11:17:32.171 INFO:teuthology.orchestra.run.smithi196.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 227 kB/s | 187 kB 00:00 2024-03-29T11:17:32.513 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 25/34 2024-03-29T11:17:32.537 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:17:32.591 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 26/34 2024-03-29T11:17:32.659 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 27/34 2024-03-29T11:17:32.689 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 28/34 2024-03-29T11:17:34.878 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:17:34.927 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:17:35.476 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 29/34 2024-03-29T11:17:35.514 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:17:35.587 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 30/34 2024-03-29T11:17:35.587 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-29T11:17:35.587 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-29T11:17:35.587 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:17:35.624 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:17:44.918 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:17:44.918 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /sys 2024-03-29T11:17:44.918 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /proc 2024-03-29T11:17:44.918 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /mnt 2024-03-29T11:17:44.918 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /var/tmp 2024-03-29T11:17:44.918 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /home 2024-03-29T11:17:44.919 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /root 2024-03-29T11:17:44.919 INFO:teuthology.orchestra.run.smithi086.stdout:skipping the directory /tmp 2024-03-29T11:17:44.919 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:17:45.836 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:17:45.860 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/34 2024-03-29T11:17:45.860 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:17:45.860 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-29T11:17:45.860 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:17:45.860 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-29T11:17:45.860 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:17:45.862 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:17:45.930 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-03-29T11:17:45.930 INFO:teuthology.orchestra.run.smithi086.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:17:47.076 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:17:47.076 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/34 2024-03-29T11:17:47.076 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/34 2024-03-29T11:17:47.076 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/34 2024-03-29T11:17:47.076 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/34 2024-03-29T11:17:47.076 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/34 2024-03-29T11:17:47.076 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_6 6/34 2024-03-29T11:17:47.076 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 7/34 2024-03-29T11:17:47.076 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9. 8/34 2024-03-29T11:17:47.076 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x8 9/34 2024-03-29T11:17:47.076 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 10/34 2024-03-29T11:17:47.076 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 11/34 2024-03-29T11:17:47.076 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 12/34 2024-03-29T11:17:47.077 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 13/34 2024-03-29T11:17:47.077 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-03-29T11:17:47.077 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-03-29T11:17:47.077 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-03-29T11:17:47.077 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-03-29T11:17:47.077 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-03-29T11:17:47.077 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-03-29T11:17:47.077 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-03-29T11:17:47.077 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-03-29T11:17:47.077 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-03-29T11:17:47.077 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 23/34 2024-03-29T11:17:47.077 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 24/34 2024-03-29T11:17:47.077 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 25/34 2024-03-29T11:17:47.077 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-03-29T11:17:47.077 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-03-29T11:17:47.077 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 28/34 2024-03-29T11:17:47.078 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : re2-1:20211101-3.el9.x86_64 29/34 2024-03-29T11:17:47.078 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-03-29T11:17:47.078 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 31/34 2024-03-29T11:17:47.078 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-03-29T11:17:47.078 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/34 2024-03-29T11:17:48.104 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-03-29T11:17:48.104 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:17:48.105 INFO:teuthology.orchestra.run.smithi086.stdout:Upgraded: 2024-03-29T11:17:48.105 INFO:teuthology.orchestra.run.smithi086.stdout: librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:48.105 INFO:teuthology.orchestra.run.smithi086.stdout: librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:48.105 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-29T11:17:48.105 INFO:teuthology.orchestra.run.smithi086.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-29T11:17:48.105 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:48.105 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:48.105 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-radosgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:48.105 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-selinux-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:48.105 INFO:teuthology.orchestra.run.smithi086.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-29T11:17:48.105 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-29T11:17:48.105 INFO:teuthology.orchestra.run.smithi086.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-29T11:17:48.105 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:48.105 INFO:teuthology.orchestra.run.smithi086.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-29T11:17:48.106 INFO:teuthology.orchestra.run.smithi086.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-29T11:17:48.106 INFO:teuthology.orchestra.run.smithi086.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-29T11:17:48.106 INFO:teuthology.orchestra.run.smithi086.stdout: libradosstriper1-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:48.106 INFO:teuthology.orchestra.run.smithi086.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-29T11:17:48.106 INFO:teuthology.orchestra.run.smithi086.stdout: librgw2-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:48.106 INFO:teuthology.orchestra.run.smithi086.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-29T11:17:48.106 INFO:teuthology.orchestra.run.smithi086.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-29T11:17:48.106 INFO:teuthology.orchestra.run.smithi086.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-29T11:17:48.106 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-argparse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:48.106 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ceph-common-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:48.106 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:48.106 INFO:teuthology.orchestra.run.smithi086.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-29T11:17:48.106 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:48.106 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:48.107 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:17:48.107 INFO:teuthology.orchestra.run.smithi086.stdout: qatlib-24.02.0-1.el9.x86_64 2024-03-29T11:17:48.107 INFO:teuthology.orchestra.run.smithi086.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-03-29T11:17:48.107 INFO:teuthology.orchestra.run.smithi086.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-03-29T11:17:48.107 INFO:teuthology.orchestra.run.smithi086.stdout: re2-1:20211101-3.el9.x86_64 2024-03-29T11:17:48.107 INFO:teuthology.orchestra.run.smithi086.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-29T11:17:48.107 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:17:48.107 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T11:17:48.476 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-test 2024-03-29T11:17:49.035 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:08 ago on Fri 29 Mar 2024 11:16:41 AM UTC. 2024-03-29T11:17:49.191 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T11:17:49.192 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:17:49.192 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-29T11:17:49.192 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:17:49.192 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-29T11:17:49.193 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 47 M 2024-03-29T11:17:49.193 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-29T11:17:49.193 INFO:teuthology.orchestra.run.smithi086.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-29T11:17:49.193 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 163 k 2024-03-29T11:17:49.193 INFO:teuthology.orchestra.run.smithi086.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-29T11:17:49.193 INFO:teuthology.orchestra.run.smithi086.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-29T11:17:49.193 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-29T11:17:49.193 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:17:49.193 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-29T11:17:49.193 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:17:49.193 INFO:teuthology.orchestra.run.smithi086.stdout:Install 6 Packages 2024-03-29T11:17:49.193 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:17:49.194 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 48 M 2024-03-29T11:17:49.194 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 204 M 2024-03-29T11:17:49.194 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-29T11:17:49.393 INFO:teuthology.orchestra.run.smithi086.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.3 MB/s | 217 kB 00:00 2024-03-29T11:17:49.435 INFO:teuthology.orchestra.run.smithi086.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.1 MB/s | 300 kB 00:00 2024-03-29T11:17:49.485 INFO:teuthology.orchestra.run.smithi086.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-29T11:17:49.710 INFO:teuthology.orchestra.run.smithi086.stdout:(4/6): libcephsqlite-19.0.0-1439.g5dcb190d.el9. 340 kB/s | 163 kB 00:00 2024-03-29T11:17:49.735 INFO:teuthology.orchestra.run.smithi086.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 748 kB/s | 187 kB 00:00 2024-03-29T11:17:54.135 INFO:teuthology.orchestra.run.smithi086.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 9.6 MB/s | 47 MB 00:04 2024-03-29T11:17:54.139 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:17:54.139 INFO:teuthology.orchestra.run.smithi086.stdout:Total 9.7 MB/s | 48 MB 00:04 2024-03-29T11:17:54.154 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-29T11:17:54.175 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-29T11:17:54.175 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-29T11:17:54.557 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-29T11:17:54.557 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-29T11:17:55.122 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-29T11:17:55.193 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:17:55.238 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:17:55.277 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:17:55.318 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:17:55.347 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:17:59.840 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:17:59.922 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:18:01.025 INFO:teuthology.orchestra.run.smithi196.stdout:(6/6): ceph-test-19.0.0-1439.g5dcb190d.el9.x86_ 1.6 MB/s | 47 MB 00:29 2024-03-29T11:18:01.028 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:01.028 INFO:teuthology.orchestra.run.smithi196.stdout:Total 1.6 MB/s | 48 MB 00:30 2024-03-29T11:18:01.042 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-29T11:18:01.064 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-29T11:18:01.064 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-29T11:18:01.440 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-29T11:18:01.440 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-29T11:18:02.023 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-29T11:18:02.108 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-29T11:18:02.159 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-29T11:18:02.199 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:18:02.247 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-29T11:18:02.278 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:18:02.372 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:18:02.373 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:18:02.373 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:18:02.373 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:18:02.373 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:18:02.373 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:18:02.753 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:18:02.753 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:02.753 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-29T11:18:02.753 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:02.753 INFO:teuthology.orchestra.run.smithi086.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:18:02.754 INFO:teuthology.orchestra.run.smithi086.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:02.754 INFO:teuthology.orchestra.run.smithi086.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:18:02.754 INFO:teuthology.orchestra.run.smithi086.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:18:02.754 INFO:teuthology.orchestra.run.smithi086.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:18:02.754 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:02.754 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T11:18:03.054 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph 2024-03-29T11:18:03.623 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:22 ago on Fri 29 Mar 2024 11:16:41 AM UTC. 2024-03-29T11:18:03.785 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T11:18:03.788 INFO:teuthology.orchestra.run.smithi086.stdout:========================================================================================= 2024-03-29T11:18:03.788 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-29T11:18:03.788 INFO:teuthology.orchestra.run.smithi086.stdout:========================================================================================= 2024-03-29T11:18:03.788 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-29T11:18:03.788 INFO:teuthology.orchestra.run.smithi086.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:18:03.788 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-29T11:18:03.788 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:18:03.788 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:18:03.788 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:18:03.789 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:18:03.789 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:18:03.789 INFO:teuthology.orchestra.run.smithi086.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:18:03.789 INFO:teuthology.orchestra.run.smithi086.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:18:03.789 INFO:teuthology.orchestra.run.smithi086.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:18:03.789 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:18:03.789 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:18:03.789 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:18:03.789 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:18:03.789 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:18:03.789 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:18:03.790 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:18:03.790 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:18:03.790 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:18:03.790 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:18:03.790 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:18:03.790 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:18:03.790 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:18:03.790 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:18:03.790 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:18:03.790 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:18:03.790 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:18:03.790 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:18:03.791 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:18:03.791 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:18:03.791 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:18:03.791 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:18:03.791 INFO:teuthology.orchestra.run.smithi086.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:18:03.791 INFO:teuthology.orchestra.run.smithi086.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:18:03.791 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:18:03.791 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:18:03.791 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:18:03.791 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:03.791 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-29T11:18:03.791 INFO:teuthology.orchestra.run.smithi086.stdout:========================================================================================= 2024-03-29T11:18:03.792 INFO:teuthology.orchestra.run.smithi086.stdout:Install 36 Packages 2024-03-29T11:18:03.792 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:03.792 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 30 M 2024-03-29T11:18:03.792 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 107 M 2024-03-29T11:18:03.792 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-29T11:18:04.497 INFO:teuthology.orchestra.run.smithi086.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 18 kB/s | 6.6 kB 00:00 2024-03-29T11:18:05.372 INFO:teuthology.orchestra.run.smithi086.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 1.2 MB/s | 1.5 MB 00:01 2024-03-29T11:18:05.581 INFO:teuthology.orchestra.run.smithi086.stdout:(3/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 1.5 MB/s | 2.1 MB 00:01 2024-03-29T11:18:05.714 INFO:teuthology.orchestra.run.smithi086.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 1.8 MB/s | 247 kB 00:00 2024-03-29T11:18:05.940 INFO:teuthology.orchestra.run.smithi086.stdout:(5/36): lua-5.4.4-4.el9.x86_64.rpm 830 kB/s | 187 kB 00:00 2024-03-29T11:18:06.048 INFO:teuthology.orchestra.run.smithi086.stdout:(6/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 3.0 MB/s | 4.7 MB 00:01 2024-03-29T11:18:06.073 INFO:teuthology.orchestra.run.smithi086.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 144 kB 00:00 2024-03-29T11:18:06.107 INFO:teuthology.orchestra.run.smithi086.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 542 kB/s | 32 kB 00:00 2024-03-29T11:18:06.132 INFO:teuthology.orchestra.run.smithi086.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 807 kB/s | 47 kB 00:00 2024-03-29T11:18:06.158 INFO:teuthology.orchestra.run.smithi086.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 770 kB/s | 39 kB 00:00 2024-03-29T11:18:06.183 INFO:teuthology.orchestra.run.smithi086.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-29T11:18:06.241 INFO:teuthology.orchestra.run.smithi086.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-29T11:18:06.267 INFO:teuthology.orchestra.run.smithi086.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-29T11:18:06.308 INFO:teuthology.orchestra.run.smithi086.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 1.8 MB/s | 124 kB 00:00 2024-03-29T11:18:06.401 INFO:teuthology.orchestra.run.smithi086.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 1.4 MB/s | 187 kB 00:00 2024-03-29T11:18:06.451 INFO:teuthology.orchestra.run.smithi086.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 446 kB/s | 22 kB 00:00 2024-03-29T11:18:06.534 INFO:teuthology.orchestra.run.smithi086.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 553 kB/s | 125 kB 00:00 2024-03-29T11:18:06.635 INFO:teuthology.orchestra.run.smithi086.stdout:(18/36): luarocks-3.9.2-1.el9.noarch.rpm 822 kB/s | 151 kB 00:00 2024-03-29T11:18:06.660 INFO:teuthology.orchestra.run.smithi086.stdout:(19/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 350 kB/s | 43 kB 00:00 2024-03-29T11:18:06.685 INFO:teuthology.orchestra.run.smithi086.stdout:(20/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 3.3 MB/s | 172 kB 00:00 2024-03-29T11:18:06.727 INFO:teuthology.orchestra.run.smithi086.stdout:(21/36): python3-cherrypy-18.6.1-2.el9.noarch.r 5.2 MB/s | 358 kB 00:00 2024-03-29T11:18:06.744 INFO:teuthology.orchestra.run.smithi086.stdout:(22/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 184 kB/s | 11 kB 00:00 2024-03-29T11:18:06.761 INFO:teuthology.orchestra.run.smithi086.stdout:(23/36): python3-jaraco-classes-3.2.1-5.el9.noa 526 kB/s | 18 kB 00:00 2024-03-29T11:18:06.778 INFO:teuthology.orchestra.run.smithi086.stdout:(24/36): python3-jaraco-collections-3.0.0-8.el9 692 kB/s | 23 kB 00:00 2024-03-29T11:18:06.803 INFO:teuthology.orchestra.run.smithi086.stdout:(25/36): python3-jaraco-functools-3.5.0-2.el9.n 462 kB/s | 19 kB 00:00 2024-03-29T11:18:06.818 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/6 2024-03-29T11:18:06.820 INFO:teuthology.orchestra.run.smithi086.stdout:(26/36): python3-jaraco-text-3.2.0-6.el9.noarch 468 kB/s | 20 kB 00:00 2024-03-29T11:18:06.846 INFO:teuthology.orchestra.run.smithi086.stdout:(27/36): python3-logutils-0.3.5-21.el9.noarch.r 1.1 MB/s | 46 kB 00:00 2024-03-29T11:18:06.871 INFO:teuthology.orchestra.run.smithi086.stdout:(28/36): python3-more-itertools-8.12.0-2.el9.no 1.5 MB/s | 79 kB 00:00 2024-03-29T11:18:06.896 INFO:teuthology.orchestra.run.smithi086.stdout:(29/36): python3-pecan-1.4.2-3.el9.noarch.rpm 5.3 MB/s | 272 kB 00:00 2024-03-29T11:18:06.901 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:18:06.922 INFO:teuthology.orchestra.run.smithi086.stdout:(30/36): python3-portend-3.1.0-2.el9.noarch.rpm 331 kB/s | 16 kB 00:00 2024-03-29T11:18:06.938 INFO:teuthology.orchestra.run.smithi086.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-03-29T11:18:06.974 INFO:teuthology.orchestra.run.smithi086.stdout:(32/36): python3-tempora-5.0.0-2.el9.noarch.rpm 686 kB/s | 36 kB 00:00 2024-03-29T11:18:06.999 INFO:teuthology.orchestra.run.smithi086.stdout:(33/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.7 MB/s | 230 kB 00:00 2024-03-29T11:18:07.050 INFO:teuthology.orchestra.run.smithi086.stdout:(34/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.6 MB/s | 427 kB 00:00 2024-03-29T11:18:07.075 INFO:teuthology.orchestra.run.smithi086.stdout:(35/36): python3-zc-lockfile-2.0-10.el9.noarch. 268 kB/s | 20 kB 00:00 2024-03-29T11:18:07.242 INFO:teuthology.orchestra.run.smithi086.stdout:(36/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 9.0 MB/s | 17 MB 00:01 2024-03-29T11:18:07.246 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:07.246 INFO:teuthology.orchestra.run.smithi086.stdout:Total 8.6 MB/s | 30 MB 00:03 2024-03-29T11:18:07.271 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-29T11:18:07.311 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-29T11:18:07.311 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-29T11:18:07.724 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-29T11:18:07.724 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-29T11:18:09.085 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-29T11:18:09.136 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:18:09.165 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:18:09.197 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:18:09.226 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 6/6 2024-03-29T11:18:09.226 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/6 2024-03-29T11:18:09.226 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/6 2024-03-29T11:18:09.226 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-29T11:18:09.226 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-29T11:18:09.226 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-29T11:18:09.267 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:18:09.312 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:18:09.345 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:18:09.439 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:18:09.523 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:18:09.564 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:18:09.625 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-29T11:18:09.625 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:18:09.625 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-29T11:18:09.625 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-test-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:09.625 INFO:teuthology.orchestra.run.smithi196.stdout: jq-1.6-16.el9.x86_64 2024-03-29T11:18:09.626 INFO:teuthology.orchestra.run.smithi196.stdout: libcephsqlite-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:09.626 INFO:teuthology.orchestra.run.smithi196.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-29T11:18:09.626 INFO:teuthology.orchestra.run.smithi196.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-29T11:18:09.626 INFO:teuthology.orchestra.run.smithi196.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-29T11:18:09.626 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:18:09.626 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T11:18:09.635 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:18:09.707 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:18:09.745 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:18:09.793 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:18:09.848 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:18:09.935 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph 2024-03-29T11:18:10.033 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:18:10.082 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:18:10.121 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:18:10.155 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:18:10.185 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:18:10.261 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:18:10.295 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:18:10.328 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:18:10.368 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:18:10.404 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:18:10.438 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:18:10.509 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:01:31 ago on Fri 29 Mar 2024 11:16:39 AM UTC. 2024-03-29T11:18:10.526 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:18:10.569 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:18:10.641 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:18:10.666 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T11:18:10.668 INFO:teuthology.orchestra.run.smithi196.stdout:========================================================================================= 2024-03-29T11:18:10.668 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-29T11:18:10.668 INFO:teuthology.orchestra.run.smithi196.stdout:========================================================================================= 2024-03-29T11:18:10.669 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-29T11:18:10.669 INFO:teuthology.orchestra.run.smithi196.stdout: ceph x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 6.6 k 2024-03-29T11:18:10.669 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-29T11:18:10.669 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mds x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 2.1 M 2024-03-29T11:18:10.669 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 1.5 M 2024-03-29T11:18:10.669 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 247 k 2024-03-29T11:18:10.669 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mon x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 4.7 M 2024-03-29T11:18:10.669 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-osd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 17 M 2024-03-29T11:18:10.669 INFO:teuthology.orchestra.run.smithi196.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-29T11:18:10.669 INFO:teuthology.orchestra.run.smithi196.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-29T11:18:10.669 INFO:teuthology.orchestra.run.smithi196.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-29T11:18:10.669 INFO:teuthology.orchestra.run.smithi196.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-29T11:18:10.669 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-29T11:18:10.669 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-29T11:18:10.670 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-29T11:18:10.670 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-29T11:18:10.670 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-29T11:18:10.670 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-29T11:18:10.670 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-29T11:18:10.670 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-29T11:18:10.670 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-29T11:18:10.670 INFO:teuthology.orchestra.run.smithi196.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-29T11:18:10.670 INFO:teuthology.orchestra.run.smithi196.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-29T11:18:10.670 INFO:teuthology.orchestra.run.smithi196.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-29T11:18:10.670 INFO:teuthology.orchestra.run.smithi196.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-29T11:18:10.670 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-29T11:18:10.670 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-29T11:18:10.670 INFO:teuthology.orchestra.run.smithi196.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-29T11:18:10.671 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-29T11:18:10.671 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-29T11:18:10.671 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-29T11:18:10.671 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-29T11:18:10.671 INFO:teuthology.orchestra.run.smithi196.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-29T11:18:10.671 INFO:teuthology.orchestra.run.smithi196.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-29T11:18:10.671 INFO:teuthology.orchestra.run.smithi196.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-29T11:18:10.671 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-29T11:18:10.671 INFO:teuthology.orchestra.run.smithi196.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-29T11:18:10.671 INFO:teuthology.orchestra.run.smithi196.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-29T11:18:10.671 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:18:10.671 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-29T11:18:10.671 INFO:teuthology.orchestra.run.smithi196.stdout:========================================================================================= 2024-03-29T11:18:10.671 INFO:teuthology.orchestra.run.smithi196.stdout:Install 36 Packages 2024-03-29T11:18:10.671 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:18:10.672 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 30 M 2024-03-29T11:18:10.672 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 107 M 2024-03-29T11:18:10.672 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-29T11:18:10.778 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:18:10.903 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:18:11.062 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:18:11.090 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:18:11.115 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:18:11.115 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:11.115 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:18:11.115 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:18:11.115 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:18:11.115 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:11.160 INFO:teuthology.orchestra.run.smithi196.stdout:(1/36): ceph-19.0.0-1439.g5dcb190d.el9.x86_64.r 25 kB/s | 6.6 kB 00:00 2024-03-29T11:18:12.533 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:18:12.559 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:18:12.560 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:12.560 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:18:12.560 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:18:12.560 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:18:12.560 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:12.802 INFO:teuthology.orchestra.run.smithi196.stdout:(2/36): ceph-mgr-19.0.0-1439.g5dcb190d.el9.x86_ 798 kB/s | 1.5 MB 00:01 2024-03-29T11:18:13.000 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:18:13.024 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:18:13.024 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:13.024 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:18:13.024 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:18:13.024 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:18:13.024 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:13.185 INFO:teuthology.orchestra.run.smithi196.stdout:(3/36): ceph-mon-19.0.0-1439.g5dcb190d.el9.x86_ 2.3 MB/s | 4.7 MB 00:02 2024-03-29T11:18:13.234 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:18:13.252 INFO:teuthology.orchestra.run.smithi196.stdout:(4/36): ceph-mgr-modules-core-19.0.0-1439.g5dcb 3.6 MB/s | 247 kB 00:00 2024-03-29T11:18:13.257 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:18:13.257 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:13.257 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:18:13.257 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:18:13.257 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:18:13.257 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:13.323 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:18:13.361 INFO:teuthology.orchestra.run.smithi196.stdout:(5/36): ceph-mds-19.0.0-1439.g5dcb190d.el9.x86_ 880 kB/s | 2.1 MB 00:02 2024-03-29T11:18:13.528 INFO:teuthology.orchestra.run.smithi196.stdout:(6/36): lua-5.4.4-4.el9.x86_64.rpm 678 kB/s | 187 kB 00:00 2024-03-29T11:18:13.603 INFO:teuthology.orchestra.run.smithi196.stdout:(7/36): python3-mako-1.1.4-6.el9.noarch.rpm 599 kB/s | 144 kB 00:00 2024-03-29T11:18:13.628 INFO:teuthology.orchestra.run.smithi196.stdout:(8/36): python3-markupsafe-1.1.1-12.el9.x86_64. 315 kB/s | 32 kB 00:00 2024-03-29T11:18:13.679 INFO:teuthology.orchestra.run.smithi196.stdout:(9/36): python3-pytz-2021.1-5.el9.noarch.rpm 629 kB/s | 47 kB 00:00 2024-03-29T11:18:13.704 INFO:teuthology.orchestra.run.smithi196.stdout:(10/36): python3-toml-0.10.2-6.el9.noarch.rpm 514 kB/s | 39 kB 00:00 2024-03-29T11:18:13.821 INFO:teuthology.orchestra.run.smithi196.stdout:(11/36): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.7 MB/s | 241 kB 00:00 2024-03-29T11:18:14.088 INFO:teuthology.orchestra.run.smithi196.stdout:(12/36): python3-cryptography-36.0.1-4.el9.x86_ 3.0 MB/s | 1.1 MB 00:00 2024-03-29T11:18:14.113 INFO:teuthology.orchestra.run.smithi196.stdout:(13/36): python3-ply-3.11-14.el9.noarch.rpm 353 kB/s | 103 kB 00:00 2024-03-29T11:18:14.138 INFO:teuthology.orchestra.run.smithi196.stdout:(14/36): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-29T11:18:14.172 INFO:teuthology.orchestra.run.smithi196.stdout:(15/36): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 187 kB 00:00 2024-03-29T11:18:14.297 INFO:teuthology.orchestra.run.smithi196.stdout:(16/36): lua-devel-5.4.4-4.el9.x86_64.rpm 178 kB/s | 22 kB 00:00 2024-03-29T11:18:14.406 INFO:teuthology.orchestra.run.smithi196.stdout:(17/36): python3-requests-2.25.1-8.el9.noarch.r 467 kB/s | 125 kB 00:00 2024-03-29T11:18:15.043 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:18:15.043 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:18:15.044 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:18:15.044 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:18:15.044 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:18:15.044 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:18:15.044 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:18:15.044 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:18:15.044 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:18:15.044 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:18:15.044 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:18:15.044 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:18:15.044 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:18:15.044 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:18:15.044 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:18:15.044 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:18:15.044 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:18:15.045 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:18:15.045 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:18:15.045 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:18:15.045 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:18:15.045 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:18:15.045 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:18:15.045 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:18:15.045 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:18:15.045 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:18:15.045 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:18:15.045 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:18:15.046 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:18:15.046 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:18:15.046 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:18:15.046 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:18:15.047 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:18:15.047 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:18:15.047 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:18:15.047 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:18:15.962 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:18:15.962 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:15.962 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-29T11:18:15.962 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:15.962 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:15.962 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:15.962 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:15.962 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:15.962 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:15.963 INFO:teuthology.orchestra.run.smithi086.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:18:15.963 INFO:teuthology.orchestra.run.smithi086.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:18:15.963 INFO:teuthology.orchestra.run.smithi086.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:18:15.963 INFO:teuthology.orchestra.run.smithi086.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:18:15.963 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:18:15.963 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:18:15.963 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:18:15.963 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:18:15.963 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:18:15.963 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:18:15.963 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:18:15.963 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:18:15.963 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:18:15.964 INFO:teuthology.orchestra.run.smithi086.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:18:15.964 INFO:teuthology.orchestra.run.smithi086.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:18:15.964 INFO:teuthology.orchestra.run.smithi086.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:18:15.964 INFO:teuthology.orchestra.run.smithi086.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:18:15.964 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:18:15.964 INFO:teuthology.orchestra.run.smithi086.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:18:15.964 INFO:teuthology.orchestra.run.smithi086.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:18:15.964 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:18:15.964 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:18:15.964 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:18:15.964 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:18:15.964 INFO:teuthology.orchestra.run.smithi086.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:18:15.964 INFO:teuthology.orchestra.run.smithi086.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:18:15.964 INFO:teuthology.orchestra.run.smithi086.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:18:15.965 INFO:teuthology.orchestra.run.smithi086.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:18:15.965 INFO:teuthology.orchestra.run.smithi086.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:18:15.965 INFO:teuthology.orchestra.run.smithi086.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:18:15.965 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:15.965 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T11:18:16.276 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-base 2024-03-29T11:18:16.869 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:35 ago on Fri 29 Mar 2024 11:16:41 AM UTC. 2024-03-29T11:18:16.974 INFO:teuthology.orchestra.run.smithi086.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:17.029 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T11:18:17.030 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-29T11:18:17.030 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T11:18:17.101 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install cephadm 2024-03-29T11:18:17.631 INFO:teuthology.orchestra.run.smithi196.stdout:(18/36): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 13 kB/s | 43 kB 00:03 2024-03-29T11:18:17.686 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:36 ago on Fri 29 Mar 2024 11:16:41 AM UTC. 2024-03-29T11:18:17.739 INFO:teuthology.orchestra.run.smithi196.stdout:(19/36): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.6 MB/s | 172 kB 00:00 2024-03-29T11:18:17.836 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T11:18:17.837 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:18:17.837 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-29T11:18:17.837 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:18:17.837 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-29T11:18:17.837 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:18:17.837 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:17.837 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-29T11:18:17.838 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:18:17.838 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-29T11:18:17.838 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:17.838 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 747 k 2024-03-29T11:18:17.838 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 754 k 2024-03-29T11:18:17.838 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-29T11:18:17.839 INFO:teuthology.orchestra.run.smithi196.stdout:(20/36): python3-cherrypy-18.6.1-2.el9.noarch.r 3.5 MB/s | 358 kB 00:00 2024-03-29T11:18:17.890 INFO:teuthology.orchestra.run.smithi196.stdout:(21/36): python3-jaraco-8.2.1-3.el9.noarch.rpm 214 kB/s | 11 kB 00:00 2024-03-29T11:18:17.940 INFO:teuthology.orchestra.run.smithi196.stdout:(22/36): python3-jaraco-classes-3.2.1-5.el9.noa 353 kB/s | 18 kB 00:00 2024-03-29T11:18:17.990 INFO:teuthology.orchestra.run.smithi196.stdout:(23/36): python3-jaraco-collections-3.0.0-8.el9 465 kB/s | 23 kB 00:00 2024-03-29T11:18:18.041 INFO:teuthology.orchestra.run.smithi196.stdout:(24/36): python3-jaraco-functools-3.5.0-2.el9.n 389 kB/s | 19 kB 00:00 2024-03-29T11:18:18.091 INFO:teuthology.orchestra.run.smithi196.stdout:(25/36): python3-jaraco-text-3.2.0-6.el9.noarch 393 kB/s | 20 kB 00:00 2024-03-29T11:18:18.130 INFO:teuthology.orchestra.run.smithi086.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 2.5 MB/s | 747 kB 00:00 2024-03-29T11:18:18.131 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:18.131 INFO:teuthology.orchestra.run.smithi086.stdout:Total 2.5 MB/s | 747 kB 00:00 2024-03-29T11:18:18.131 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-29T11:18:18.136 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-29T11:18:18.136 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-29T11:18:18.141 INFO:teuthology.orchestra.run.smithi196.stdout:(26/36): python3-logutils-0.3.5-21.el9.noarch.r 928 kB/s | 46 kB 00:00 2024-03-29T11:18:18.142 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-29T11:18:18.142 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-29T11:18:18.208 INFO:teuthology.orchestra.run.smithi196.stdout:(27/36): python3-more-itertools-8.12.0-2.el9.no 1.2 MB/s | 79 kB 00:00 2024-03-29T11:18:18.216 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-29T11:18:18.292 INFO:teuthology.orchestra.run.smithi196.stdout:(28/36): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-03-29T11:18:18.351 INFO:teuthology.orchestra.run.smithi196.stdout:(29/36): python3-portend-3.1.0-2.el9.noarch.rpm 281 kB/s | 16 kB 00:00 2024-03-29T11:18:18.551 INFO:teuthology.orchestra.run.smithi196.stdout:(30/36): ceph-osd-19.0.0-1439.g5dcb190d.el9.x86 2.9 MB/s | 17 MB 00:05 2024-03-29T11:18:18.576 INFO:teuthology.orchestra.run.smithi196.stdout:(31/36): python3-pyOpenSSL-21.0.0-1.el9.noarch. 399 kB/s | 90 kB 00:00 2024-03-29T11:18:18.643 INFO:teuthology.orchestra.run.smithi196.stdout:(32/36): python3-webob-1.8.7-6.el9.noarch.rpm 3.4 MB/s | 230 kB 00:00 2024-03-29T11:18:18.663 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:18:18.727 INFO:teuthology.orchestra.run.smithi196.stdout:(33/36): python3-werkzeug-2.0.3-3.el9.1.noarch. 5.0 MB/s | 427 kB 00:00 2024-03-29T11:18:18.743 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:18:18.785 INFO:teuthology.orchestra.run.smithi196.stdout:(34/36): python3-zc-lockfile-2.0-10.el9.noarch. 342 kB/s | 20 kB 00:00 2024-03-29T11:18:19.107 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:18:19.385 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:18:19.385 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:19.385 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-29T11:18:19.385 INFO:teuthology.orchestra.run.smithi086.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:19.385 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:19.385 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T11:18:19.584 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:18:19.694 INFO:teuthology.orchestra.run.smithi196.stdout:(35/36): python3-tempora-5.0.0-2.el9.noarch.rpm 31 kB/s | 36 kB 00:01 2024-03-29T11:18:20.158 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:39 ago on Fri 29 Mar 2024 11:16:41 AM UTC. 2024-03-29T11:18:20.307 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T11:18:20.308 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:18:20.308 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-29T11:18:20.309 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:18:20.309 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-29T11:18:20.309 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:18:20.309 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:20.309 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-29T11:18:20.309 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:18:20.309 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-29T11:18:20.309 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:20.309 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 145 k 2024-03-29T11:18:20.309 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 436 k 2024-03-29T11:18:20.309 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-29T11:18:20.677 INFO:teuthology.orchestra.run.smithi086.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 395 kB/s | 145 kB 00:00 2024-03-29T11:18:20.678 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:20.678 INFO:teuthology.orchestra.run.smithi086.stdout:Total 394 kB/s | 145 kB 00:00 2024-03-29T11:18:20.678 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-29T11:18:20.686 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-29T11:18:20.686 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-29T11:18:20.739 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-29T11:18:20.740 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-29T11:18:20.880 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-29T11:18:20.945 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:18:20.970 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:18:20.970 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:20.970 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:18:20.970 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:21.934 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:18:21.934 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:21.934 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-29T11:18:21.934 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:21.934 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:21.935 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T11:18:22.171 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr 2024-03-29T11:18:22.748 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:41 ago on Fri 29 Mar 2024 11:16:41 AM UTC. 2024-03-29T11:18:22.846 INFO:teuthology.orchestra.run.smithi086.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:22.898 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T11:18:22.899 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-29T11:18:22.899 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T11:18:22.968 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:18:23.526 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:42 ago on Fri 29 Mar 2024 11:16:41 AM UTC. 2024-03-29T11:18:23.677 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T11:18:23.678 INFO:teuthology.orchestra.run.smithi086.stdout:====================================================================================== 2024-03-29T11:18:23.678 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-29T11:18:23.678 INFO:teuthology.orchestra.run.smithi086.stdout:====================================================================================== 2024-03-29T11:18:23.678 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-29T11:18:23.678 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:18:23.678 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-29T11:18:23.678 INFO:teuthology.orchestra.run.smithi086.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:18:23.679 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:18:23.679 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:18:23.679 INFO:teuthology.orchestra.run.smithi086.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:18:23.679 INFO:teuthology.orchestra.run.smithi086.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:18:23.679 INFO:teuthology.orchestra.run.smithi086.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:18:23.679 INFO:teuthology.orchestra.run.smithi086.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:18:23.679 INFO:teuthology.orchestra.run.smithi086.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:18:23.679 INFO:teuthology.orchestra.run.smithi086.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:18:23.679 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:18:23.679 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:18:23.679 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:23.679 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-29T11:18:23.680 INFO:teuthology.orchestra.run.smithi086.stdout:====================================================================================== 2024-03-29T11:18:23.680 INFO:teuthology.orchestra.run.smithi086.stdout:Install 12 Packages 2024-03-29T11:18:23.680 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:23.680 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 6.9 M 2024-03-29T11:18:23.680 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 26 M 2024-03-29T11:18:23.680 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-29T11:18:24.171 INFO:teuthology.orchestra.run.smithi086.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 51 kB/s | 15 kB 00:00 2024-03-29T11:18:24.197 INFO:teuthology.orchestra.run.smithi086.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 96 kB/s | 31 kB 00:00 2024-03-29T11:18:24.330 INFO:teuthology.orchestra.run.smithi086.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 2.1 MB/s | 238 kB 00:00 2024-03-29T11:18:24.364 INFO:teuthology.orchestra.run.smithi086.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 5.2 MB/s | 1.0 MB 00:00 2024-03-29T11:18:24.514 INFO:teuthology.orchestra.run.smithi086.stdout:(5/12): protobuf-compiler-3.14.0-13.el9.x86_64. 4.6 MB/s | 863 kB 00:00 2024-03-29T11:18:24.739 INFO:teuthology.orchestra.run.smithi086.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 87 kB/s | 19 kB 00:00 2024-03-29T11:18:24.831 INFO:teuthology.orchestra.run.smithi086.stdout:(7/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 1.1 MB/s | 548 kB 00:00 2024-03-29T11:18:24.906 INFO:teuthology.orchestra.run.smithi086.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 1.9 MB/s | 144 kB 00:00 2024-03-29T11:18:24.974 INFO:teuthology.orchestra.run.smithi086.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 461 kB/s | 31 kB 00:00 2024-03-29T11:18:25.049 INFO:teuthology.orchestra.run.smithi086.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 2.5 MB/s | 188 kB 00:00 2024-03-29T11:18:25.082 INFO:teuthology.orchestra.run.smithi086.stdout:(11/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb19 1.5 MB/s | 1.8 MB 00:01 2024-03-29T11:18:25.141 INFO:teuthology.orchestra.run.smithi086.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 5.1 MB/s | 2.0 MB 00:00 2024-03-29T11:18:25.146 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:25.146 INFO:teuthology.orchestra.run.smithi086.stdout:Total 4.7 MB/s | 6.9 MB 00:01 2024-03-29T11:18:25.164 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-29T11:18:25.180 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-29T11:18:25.180 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-29T11:18:25.354 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-29T11:18:25.354 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-29T11:18:25.861 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-29T11:18:25.941 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:18:25.977 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:18:26.024 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:18:26.054 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:18:26.158 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:18:26.278 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:18:26.373 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:18:26.416 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:18:26.444 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:18:26.476 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:18:26.720 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:18:26.751 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:18:27.609 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:18:27.610 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:18:27.610 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:18:27.610 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:18:27.610 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:18:27.610 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:18:27.611 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:18:27.611 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:18:27.611 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:18:27.611 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:18:27.611 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:18:27.611 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:18:27.978 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:18:27.978 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:27.978 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-29T11:18:27.978 INFO:teuthology.orchestra.run.smithi086.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:18:27.978 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:27.978 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:27.978 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:27.978 INFO:teuthology.orchestra.run.smithi086.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:18:27.978 INFO:teuthology.orchestra.run.smithi086.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:18:27.979 INFO:teuthology.orchestra.run.smithi086.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:18:27.979 INFO:teuthology.orchestra.run.smithi086.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:18:27.979 INFO:teuthology.orchestra.run.smithi086.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:18:27.979 INFO:teuthology.orchestra.run.smithi086.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:18:27.979 INFO:teuthology.orchestra.run.smithi086.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:18:27.979 INFO:teuthology.orchestra.run.smithi086.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:18:27.979 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:27.979 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T11:18:28.237 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:18:28.794 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:47 ago on Fri 29 Mar 2024 11:16:41 AM UTC. 2024-03-29T11:18:28.945 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T11:18:28.946 INFO:teuthology.orchestra.run.smithi086.stdout:============================================================================================ 2024-03-29T11:18:28.946 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-29T11:18:28.946 INFO:teuthology.orchestra.run.smithi086.stdout:============================================================================================ 2024-03-29T11:18:28.946 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-29T11:18:28.946 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:18:28.947 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-29T11:18:28.947 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:18:28.947 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:18:28.947 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:18:28.947 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:18:28.947 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:18:28.947 INFO:teuthology.orchestra.run.smithi086.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:18:28.947 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:18:28.947 INFO:teuthology.orchestra.run.smithi086.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:18:28.947 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:18:28.947 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:18:28.948 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:18:28.948 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:28.948 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-29T11:18:28.948 INFO:teuthology.orchestra.run.smithi086.stdout:============================================================================================ 2024-03-29T11:18:28.948 INFO:teuthology.orchestra.run.smithi086.stdout:Install 12 Packages 2024-03-29T11:18:28.948 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:28.948 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 37 M 2024-03-29T11:18:28.949 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 209 M 2024-03-29T11:18:28.949 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-29T11:18:29.172 INFO:teuthology.orchestra.run.smithi086.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 248 kB/s | 32 kB 00:00 2024-03-29T11:18:29.205 INFO:teuthology.orchestra.run.smithi086.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 534 kB/s | 17 kB 00:00 2024-03-29T11:18:29.280 INFO:teuthology.orchestra.run.smithi086.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 13 MB/s | 3.0 MB 00:00 2024-03-29T11:18:29.306 INFO:teuthology.orchestra.run.smithi086.stdout:(4/12): openblas-0.3.21-2.el9.x86_64.rpm 348 kB/s | 34 kB 00:00 2024-03-29T11:18:29.515 INFO:teuthology.orchestra.run.smithi086.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-03-29T11:18:29.581 INFO:teuthology.orchestra.run.smithi086.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 4.8 MB 00:00 2024-03-29T11:18:29.607 INFO:teuthology.orchestra.run.smithi086.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.3 MB/s | 217 kB 00:00 2024-03-29T11:18:30.007 INFO:teuthology.orchestra.run.smithi086.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-03-29T11:18:30.232 INFO:teuthology.orchestra.run.smithi086.stdout:(9/12): libquadmath-11.4.1-3.el9.x86_64.rpm 869 kB/s | 195 kB 00:00 2024-03-29T11:18:30.299 INFO:teuthology.orchestra.run.smithi086.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2024-03-29T11:18:30.891 INFO:teuthology.orchestra.run.smithi086.stdout:(11/12): ceph-mgr-diskprediction-local-19.0.0-1 4.0 MB/s | 7.4 MB 00:01 2024-03-29T11:18:31.191 INFO:teuthology.orchestra.run.smithi086.stdout:(12/12): python3-devel-3.9.17-2.el9.x86_64.rpm 256 kB/s | 245 kB 00:00 2024-03-29T11:18:31.194 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:31.194 INFO:teuthology.orchestra.run.smithi086.stdout:Total 16 MB/s | 37 MB 00:02 2024-03-29T11:18:31.238 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-29T11:18:31.264 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-29T11:18:31.264 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-29T11:18:31.636 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-29T11:18:31.636 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-29T11:18:32.354 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-29T11:18:32.439 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:18:32.479 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:18:32.574 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:18:32.615 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:18:32.793 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:18:32.814 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:18:32.890 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:18:33.576 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:18:33.701 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:18:34.939 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:18:35.975 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:18:36.004 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:18:38.427 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:18:38.427 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:18:38.427 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:18:38.428 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:18:38.428 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:18:38.428 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:18:38.428 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:18:38.428 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:18:38.428 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:18:38.428 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:18:38.428 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:18:38.428 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:18:38.855 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:18:38.855 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:38.855 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-29T11:18:38.855 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:38.855 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:18:38.855 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:18:38.855 INFO:teuthology.orchestra.run.smithi086.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:18:38.856 INFO:teuthology.orchestra.run.smithi086.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:18:38.856 INFO:teuthology.orchestra.run.smithi086.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:18:38.856 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:18:38.856 INFO:teuthology.orchestra.run.smithi086.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:18:38.856 INFO:teuthology.orchestra.run.smithi086.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:18:38.856 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:18:38.856 INFO:teuthology.orchestra.run.smithi086.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:18:38.856 INFO:teuthology.orchestra.run.smithi086.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:18:38.856 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:38.856 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T11:18:39.201 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:18:39.774 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:01:58 ago on Fri 29 Mar 2024 11:16:41 AM UTC. 2024-03-29T11:18:39.926 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T11:18:39.927 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================== 2024-03-29T11:18:39.927 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-29T11:18:39.928 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================== 2024-03-29T11:18:39.928 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-29T11:18:39.928 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:18:39.928 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-29T11:18:39.928 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:18:39.928 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:18:39.928 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:18:39.928 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:18:39.928 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:18:39.928 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:18:39.928 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:18:39.928 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:18:39.928 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:18:39.929 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:18:39.929 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:18:39.929 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:18:39.929 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:39.929 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-29T11:18:39.929 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================== 2024-03-29T11:18:39.929 INFO:teuthology.orchestra.run.smithi086.stdout:Install 13 Packages 2024-03-29T11:18:39.929 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:39.929 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 2.0 M 2024-03-29T11:18:39.930 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 26 M 2024-03-29T11:18:39.930 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-29T11:18:40.128 INFO:teuthology.orchestra.run.smithi086.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 216 kB/s | 24 kB 00:00 2024-03-29T11:18:40.153 INFO:teuthology.orchestra.run.smithi086.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 131 kB/s | 18 kB 00:00 2024-03-29T11:18:40.203 INFO:teuthology.orchestra.run.smithi086.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.3 MB/s | 174 kB 00:00 2024-03-29T11:18:40.236 INFO:teuthology.orchestra.run.smithi086.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.6 MB/s | 132 kB 00:00 2024-03-29T11:18:40.262 INFO:teuthology.orchestra.run.smithi086.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 3.5 MB/s | 211 kB 00:00 2024-03-29T11:18:40.287 INFO:teuthology.orchestra.run.smithi086.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 855 kB/s | 43 kB 00:00 2024-03-29T11:18:40.370 INFO:teuthology.orchestra.run.smithi086.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 147 kB/s | 52 kB 00:00 2024-03-29T11:18:40.546 INFO:teuthology.orchestra.run.smithi086.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 58 kB/s | 15 kB 00:00 2024-03-29T11:18:40.579 INFO:teuthology.orchestra.run.smithi086.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 101 kB/s | 32 kB 00:00 2024-03-29T11:18:40.796 INFO:teuthology.orchestra.run.smithi086.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 273 kB/s | 59 kB 00:00 2024-03-29T11:18:40.905 INFO:teuthology.orchestra.run.smithi086.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 379 kB/s | 203 kB 00:00 2024-03-29T11:18:40.983 INFO:teuthology.orchestra.run.smithi086.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 481 kB/s | 90 kB 00:00 2024-03-29T11:18:41.105 INFO:teuthology.orchestra.run.smithi086.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.8 MB/s | 1.0 MB 00:00 2024-03-29T11:18:41.107 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:41.107 INFO:teuthology.orchestra.run.smithi086.stdout:Total 1.7 MB/s | 2.0 MB 00:01 2024-03-29T11:18:41.108 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-29T11:18:41.121 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-29T11:18:41.121 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-29T11:18:41.210 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-29T11:18:41.211 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-29T11:18:41.730 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-29T11:18:41.892 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:18:41.947 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:18:42.002 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:18:42.032 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:18:42.081 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:18:42.175 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:18:42.400 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:18:42.450 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:18:43.168 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:18:43.221 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:18:43.255 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:18:43.295 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:18:43.325 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:18:44.046 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:18:44.047 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:18:44.047 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:18:44.047 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:18:44.047 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:18:44.047 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:18:44.047 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:18:44.047 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:18:44.047 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:18:44.047 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:18:44.047 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:18:44.047 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:18:44.047 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:18:44.424 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:18:44.424 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:44.424 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-29T11:18:44.424 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:44.424 INFO:teuthology.orchestra.run.smithi086.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:18:44.424 INFO:teuthology.orchestra.run.smithi086.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:18:44.424 INFO:teuthology.orchestra.run.smithi086.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:18:44.425 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:18:44.425 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:18:44.425 INFO:teuthology.orchestra.run.smithi086.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:18:44.425 INFO:teuthology.orchestra.run.smithi086.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:18:44.425 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:18:44.425 INFO:teuthology.orchestra.run.smithi086.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:18:44.425 INFO:teuthology.orchestra.run.smithi086.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:18:44.425 INFO:teuthology.orchestra.run.smithi086.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:18:44.425 INFO:teuthology.orchestra.run.smithi086.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:18:44.425 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:44.425 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T11:18:44.668 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:18:45.235 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:04 ago on Fri 29 Mar 2024 11:16:41 AM UTC. 2024-03-29T11:18:45.388 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T11:18:45.388 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================== 2024-03-29T11:18:45.388 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-29T11:18:45.389 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================== 2024-03-29T11:18:45.389 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-29T11:18:45.389 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:18:45.389 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-29T11:18:45.389 INFO:teuthology.orchestra.run.smithi086.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:18:45.389 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:18:45.389 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:18:45.389 INFO:teuthology.orchestra.run.smithi086.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:18:45.389 INFO:teuthology.orchestra.run.smithi086.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:18:45.389 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:45.390 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-29T11:18:45.390 INFO:teuthology.orchestra.run.smithi086.stdout:======================================================================================== 2024-03-29T11:18:45.390 INFO:teuthology.orchestra.run.smithi086.stdout:Install 6 Packages 2024-03-29T11:18:45.390 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:45.390 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 6.8 M 2024-03-29T11:18:45.390 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 33 M 2024-03-29T11:18:45.390 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-29T11:18:46.255 INFO:teuthology.orchestra.run.smithi086.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 19 MB/s | 5.8 MB 00:00 2024-03-29T11:18:46.356 INFO:teuthology.orchestra.run.smithi086.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 337 kB/s | 138 kB 00:00 2024-03-29T11:18:46.456 INFO:teuthology.orchestra.run.smithi196.stdout:(36/36): luarocks-3.9.2-1.el9.noarch.rpm 4.7 kB/s | 151 kB 00:32 2024-03-29T11:18:46.460 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:46.460 INFO:teuthology.orchestra.run.smithi196.stdout:Total 849 kB/s | 30 MB 00:35 2024-03-29T11:18:46.483 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-29T11:18:46.524 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-29T11:18:46.524 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-29T11:18:46.806 INFO:teuthology.orchestra.run.smithi086.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 135 kB/s | 58 kB 00:00 2024-03-29T11:18:46.925 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-29T11:18:46.925 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-29T11:18:46.939 INFO:teuthology.orchestra.run.smithi086.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 382 kB/s | 51 kB 00:00 2024-03-29T11:18:46.965 INFO:teuthology.orchestra.run.smithi086.stdout:(5/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 244 kB/s | 248 kB 00:01 2024-03-29T11:18:46.998 INFO:teuthology.orchestra.run.smithi086.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 763 kB/s | 548 kB 00:00 2024-03-29T11:18:47.001 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:47.002 INFO:teuthology.orchestra.run.smithi086.stdout:Total 4.2 MB/s | 6.8 MB 00:01 2024-03-29T11:18:47.015 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-29T11:18:47.025 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-29T11:18:47.025 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-29T11:18:47.114 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-29T11:18:47.114 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-29T11:18:47.420 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-29T11:18:47.613 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:18:47.658 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:18:48.177 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:18:48.248 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-29T11:18:48.252 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:18:48.313 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:18:48.317 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/36 2024-03-29T11:18:48.337 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:18:48.358 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/36 2024-03-29T11:18:48.386 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/36 2024-03-29T11:18:48.456 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/36 2024-03-29T11:18:48.493 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : luarocks-3.9.2-1.el9.noarch 5/36 2024-03-29T11:18:48.526 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/36 2024-03-29T11:18:48.617 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/36 2024-03-29T11:18:48.695 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/36 2024-03-29T11:18:48.737 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 9/36 2024-03-29T11:18:48.808 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/36 2024-03-29T11:18:48.871 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 11/36 2024-03-29T11:18:48.909 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 12/36 2024-03-29T11:18:48.958 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ply-3.11-14.el9.noarch 13/36 2024-03-29T11:18:49.014 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 14/36 2024-03-29T11:18:49.141 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:18:49.141 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:18:49.142 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:18:49.142 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:18:49.142 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:18:49.142 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:18:49.197 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 15/36 2024-03-29T11:18:49.255 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 16/36 2024-03-29T11:18:49.294 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 17/36 2024-03-29T11:18:49.328 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 18/36 2024-03-29T11:18:49.383 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 19/36 2024-03-29T11:18:49.451 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 20/36 2024-03-29T11:18:49.452 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:18:49.452 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:49.452 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-29T11:18:49.452 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:49.452 INFO:teuthology.orchestra.run.smithi086.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:18:49.452 INFO:teuthology.orchestra.run.smithi086.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:18:49.453 INFO:teuthology.orchestra.run.smithi086.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:18:49.453 INFO:teuthology.orchestra.run.smithi086.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:18:49.453 INFO:teuthology.orchestra.run.smithi086.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:18:49.453 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:49.453 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T11:18:49.494 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:18:49.527 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 22/36 2024-03-29T11:18:49.568 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 23/36 2024-03-29T11:18:49.603 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 24/36 2024-03-29T11:18:49.637 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 25/36 2024-03-29T11:18:49.722 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-fuse 2024-03-29T11:18:49.726 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 26/36 2024-03-29T11:18:49.768 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 27/36 2024-03-29T11:18:49.840 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 28/36 2024-03-29T11:18:49.975 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 29/36 2024-03-29T11:18:50.093 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:18:50.251 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 31/36 2024-03-29T11:18:50.273 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:18:50.287 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:09 ago on Fri 29 Mar 2024 11:16:41 AM UTC. 2024-03-29T11:18:50.297 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 32/36 2024-03-29T11:18:50.297 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:50.297 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-29T11:18:50.298 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:18:50.298 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-29T11:18:50.298 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:18:50.438 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T11:18:50.442 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:18:50.442 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-29T11:18:50.442 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:18:50.442 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-29T11:18:50.442 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:18:50.442 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-29T11:18:50.442 INFO:teuthology.orchestra.run.smithi086.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:18:50.442 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:50.442 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-29T11:18:50.443 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:18:50.443 INFO:teuthology.orchestra.run.smithi086.stdout:Install 2 Packages 2024-03-29T11:18:50.443 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:50.443 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 964 k 2024-03-29T11:18:50.443 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 2.8 M 2024-03-29T11:18:50.443 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-29T11:18:50.575 INFO:teuthology.orchestra.run.smithi086.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 588 kB/s | 79 kB 00:00 2024-03-29T11:18:50.773 INFO:teuthology.orchestra.run.smithi086.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 2.6 MB/s | 885 kB 00:00 2024-03-29T11:18:50.774 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:50.774 INFO:teuthology.orchestra.run.smithi086.stdout:Total 2.8 MB/s | 964 kB 00:00 2024-03-29T11:18:50.774 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-29T11:18:50.795 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-29T11:18:50.795 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-29T11:18:50.872 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-29T11:18:50.872 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-29T11:18:51.062 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-29T11:18:51.186 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:18:51.258 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:18:51.732 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:18:51.758 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 33/36 2024-03-29T11:18:51.759 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:51.759 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-29T11:18:51.759 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:18:51.759 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-29T11:18:51.759 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:18:51.989 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:18:51.989 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:18:52.200 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:18:52.222 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 34/36 2024-03-29T11:18:52.222 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:52.222 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-29T11:18:52.222 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:18:52.222 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-29T11:18:52.223 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:18:52.298 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:18:52.298 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:52.298 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-29T11:18:52.298 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:18:52.298 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:52.298 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T11:18:52.425 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:18:52.448 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 35/36 2024-03-29T11:18:52.448 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:52.448 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-29T11:18:52.448 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:18:52.448 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-29T11:18:52.449 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:18:52.514 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:18:52.595 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install ceph-volume 2024-03-29T11:18:53.165 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:12 ago on Fri 29 Mar 2024 11:16:41 AM UTC. 2024-03-29T11:18:53.317 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T11:18:53.318 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:18:53.318 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repository Size 2024-03-29T11:18:53.318 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:18:53.318 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-29T11:18:53.318 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:18:53.318 INFO:teuthology.orchestra.run.smithi086.stdout:Installing dependencies: 2024-03-29T11:18:53.318 INFO:teuthology.orchestra.run.smithi086.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:18:53.318 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:53.318 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-29T11:18:53.318 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:18:53.318 INFO:teuthology.orchestra.run.smithi086.stdout:Install 2 Packages 2024-03-29T11:18:53.319 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:53.319 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 332 k 2024-03-29T11:18:53.319 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 1.5 M 2024-03-29T11:18:53.319 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-29T11:18:53.438 INFO:teuthology.orchestra.run.smithi086.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 581 kB/s | 69 kB 00:00 2024-03-29T11:18:53.696 INFO:teuthology.orchestra.run.smithi086.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 700 kB/s | 264 kB 00:00 2024-03-29T11:18:53.697 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:53.697 INFO:teuthology.orchestra.run.smithi086.stdout:Total 877 kB/s | 332 kB 00:00 2024-03-29T11:18:53.697 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-29T11:18:53.706 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-29T11:18:53.706 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-29T11:18:53.746 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-29T11:18:53.746 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-29T11:18:53.893 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-29T11:18:54.068 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:18:54.106 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:18:54.125 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:18:54.125 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:18:54.125 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:18:54.125 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:54.239 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 36/36 2024-03-29T11:18:54.239 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/36 2024-03-29T11:18:54.239 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/36 2024-03-29T11:18:54.239 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 3/36 2024-03-29T11:18:54.239 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 4/36 2024-03-29T11:18:54.239 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 5/36 2024-03-29T11:18:54.239 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9. 6/36 2024-03-29T11:18:54.239 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lua-5.4.4-4.el9.x86_64 7/36 2024-03-29T11:18:54.240 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 8/36 2024-03-29T11:18:54.240 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 9/36 2024-03-29T11:18:54.240 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 10/36 2024-03-29T11:18:54.242 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 11/36 2024-03-29T11:18:54.242 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 12/36 2024-03-29T11:18:54.242 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 13/36 2024-03-29T11:18:54.242 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ply-3.11-14.el9.noarch 14/36 2024-03-29T11:18:54.242 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 15/36 2024-03-29T11:18:54.242 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 16/36 2024-03-29T11:18:54.243 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 17/36 2024-03-29T11:18:54.243 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 18/36 2024-03-29T11:18:54.243 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 19/36 2024-03-29T11:18:54.243 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 20/36 2024-03-29T11:18:54.243 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 21/36 2024-03-29T11:18:54.243 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 22/36 2024-03-29T11:18:54.243 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 23/36 2024-03-29T11:18:54.243 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 24/36 2024-03-29T11:18:54.243 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 25/36 2024-03-29T11:18:54.243 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 26/36 2024-03-29T11:18:54.243 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 27/36 2024-03-29T11:18:54.243 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 28/36 2024-03-29T11:18:54.243 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 29/36 2024-03-29T11:18:54.243 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 30/36 2024-03-29T11:18:54.243 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 31/36 2024-03-29T11:18:54.244 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 32/36 2024-03-29T11:18:54.244 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 33/36 2024-03-29T11:18:54.244 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 34/36 2024-03-29T11:18:54.244 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 35/36 2024-03-29T11:18:54.894 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:18:55.181 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 36/36 2024-03-29T11:18:55.181 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:18:55.181 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-29T11:18:55.181 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:55.181 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mds-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:55.181 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:55.181 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-modules-core-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:55.181 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mon-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:55.181 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-osd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:55.182 INFO:teuthology.orchestra.run.smithi196.stdout: lua-5.4.4-4.el9.x86_64 2024-03-29T11:18:55.182 INFO:teuthology.orchestra.run.smithi196.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-29T11:18:55.182 INFO:teuthology.orchestra.run.smithi196.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-29T11:18:55.182 INFO:teuthology.orchestra.run.smithi196.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-29T11:18:55.182 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-29T11:18:55.182 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-29T11:18:55.182 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-29T11:18:55.182 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-29T11:18:55.182 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-29T11:18:55.182 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-29T11:18:55.182 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-29T11:18:55.183 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-29T11:18:55.183 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-29T11:18:55.183 INFO:teuthology.orchestra.run.smithi196.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-29T11:18:55.183 INFO:teuthology.orchestra.run.smithi196.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-29T11:18:55.183 INFO:teuthology.orchestra.run.smithi196.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-29T11:18:55.183 INFO:teuthology.orchestra.run.smithi196.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-29T11:18:55.183 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-29T11:18:55.183 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ply-3.11-14.el9.noarch 2024-03-29T11:18:55.183 INFO:teuthology.orchestra.run.smithi196.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-29T11:18:55.183 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-29T11:18:55.183 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-29T11:18:55.183 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-29T11:18:55.184 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-29T11:18:55.184 INFO:teuthology.orchestra.run.smithi196.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-29T11:18:55.184 INFO:teuthology.orchestra.run.smithi196.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-29T11:18:55.184 INFO:teuthology.orchestra.run.smithi196.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-29T11:18:55.184 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-29T11:18:55.184 INFO:teuthology.orchestra.run.smithi196.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-29T11:18:55.184 INFO:teuthology.orchestra.run.smithi196.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-29T11:18:55.184 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:18:55.184 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T11:18:55.230 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:18:55.231 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:55.231 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-29T11:18:55.231 INFO:teuthology.orchestra.run.smithi086.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:55.231 INFO:teuthology.orchestra.run.smithi086.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:18:55.231 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:55.231 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T11:18:55.442 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-base 2024-03-29T11:18:55.575 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados-devel 2024-03-29T11:18:56.006 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:02:17 ago on Fri 29 Mar 2024 11:16:39 AM UTC. 2024-03-29T11:18:56.105 INFO:teuthology.orchestra.run.smithi196.stdout:Package ceph-base-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:56.150 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:15 ago on Fri 29 Mar 2024 11:16:41 AM UTC. 2024-03-29T11:18:56.158 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T11:18:56.159 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-29T11:18:56.159 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T11:18:56.225 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install cephadm 2024-03-29T11:18:56.299 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T11:18:56.300 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:18:56.300 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-29T11:18:56.300 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:18:56.300 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-29T11:18:56.300 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:18:56.300 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:56.300 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-29T11:18:56.300 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:18:56.300 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-29T11:18:56.300 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:56.300 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 127 k 2024-03-29T11:18:56.300 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 456 k 2024-03-29T11:18:56.300 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-29T11:18:56.676 INFO:teuthology.orchestra.run.smithi086.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 338 kB/s | 127 kB 00:00 2024-03-29T11:18:56.677 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:56.677 INFO:teuthology.orchestra.run.smithi086.stdout:Total 336 kB/s | 127 kB 00:00 2024-03-29T11:18:56.677 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-29T11:18:56.683 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-29T11:18:56.683 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-29T11:18:56.739 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-29T11:18:56.739 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-29T11:18:56.787 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:02:17 ago on Fri 29 Mar 2024 11:16:39 AM UTC. 2024-03-29T11:18:56.881 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-29T11:18:56.936 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T11:18:56.937 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:18:56.937 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-29T11:18:56.937 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:18:56.937 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-29T11:18:56.937 INFO:teuthology.orchestra.run.smithi196.stdout: cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 747 k 2024-03-29T11:18:56.937 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:18:56.937 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-29T11:18:56.937 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:18:56.937 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-29T11:18:56.938 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:18:56.938 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 747 k 2024-03-29T11:18:56.938 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 754 k 2024-03-29T11:18:56.938 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-29T11:18:56.992 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:57.493 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:57.643 INFO:teuthology.orchestra.run.smithi196.stdout:cephadm-19.0.0-1439.g5dcb190d.el9.noarch.rpm 1.0 MB/s | 747 kB 00:00 2024-03-29T11:18:57.644 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:57.644 INFO:teuthology.orchestra.run.smithi196.stdout:Total 1.0 MB/s | 747 kB 00:00 2024-03-29T11:18:57.644 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-29T11:18:57.650 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-29T11:18:57.650 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-29T11:18:57.656 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-29T11:18:57.656 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-29T11:18:57.732 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-29T11:18:57.757 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:18:57.757 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:57.758 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-29T11:18:57.758 INFO:teuthology.orchestra.run.smithi086.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:18:57.758 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:57.758 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T11:18:58.003 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install libcephfs2 2024-03-29T11:18:58.180 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:18:58.274 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:18:58.575 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:17 ago on Fri 29 Mar 2024 11:16:41 AM UTC. 2024-03-29T11:18:58.626 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:18:58.677 INFO:teuthology.orchestra.run.smithi086.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:18:58.731 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T11:18:58.732 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-29T11:18:58.732 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T11:18:58.800 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install libcephfs-devel 2024-03-29T11:18:58.902 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/1 2024-03-29T11:18:58.902 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:18:58.902 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-29T11:18:58.902 INFO:teuthology.orchestra.run.smithi196.stdout: cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:18:58.902 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:18:58.902 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T11:18:59.157 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-immutable-object-cache 2024-03-29T11:18:59.368 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:18 ago on Fri 29 Mar 2024 11:16:41 AM UTC. 2024-03-29T11:18:59.519 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T11:18:59.520 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:18:59.520 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-29T11:18:59.520 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:18:59.520 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-29T11:18:59.520 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:18:59.520 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:59.520 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-29T11:18:59.521 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:18:59.521 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-29T11:18:59.521 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:18:59.521 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 31 k 2024-03-29T11:18:59.521 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 140 k 2024-03-29T11:18:59.521 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-29T11:18:59.687 INFO:teuthology.orchestra.run.smithi086.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 188 kB/s | 31 kB 00:00 2024-03-29T11:18:59.687 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:18:59.687 INFO:teuthology.orchestra.run.smithi086.stdout:Total 186 kB/s | 31 kB 00:00 2024-03-29T11:18:59.688 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-29T11:18:59.691 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-29T11:18:59.691 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-29T11:18:59.729 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-29T11:18:59.729 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-29T11:18:59.732 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:02:20 ago on Fri 29 Mar 2024 11:16:39 AM UTC. 2024-03-29T11:18:59.868 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-29T11:18:59.888 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T11:18:59.889 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:18:59.889 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-29T11:18:59.889 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:18:59.889 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-29T11:18:59.889 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 145 k 2024-03-29T11:18:59.889 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:18:59.889 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-29T11:18:59.889 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:18:59.889 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-29T11:18:59.889 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:18:59.890 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 145 k 2024-03-29T11:18:59.890 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 436 k 2024-03-29T11:18:59.890 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-29T11:18:59.993 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:00.108 INFO:teuthology.orchestra.run.smithi196.stdout:ceph-immutable-object-cache-19.0.0-1439.g5dcb19 670 kB/s | 145 kB 00:00 2024-03-29T11:19:00.108 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:00.109 INFO:teuthology.orchestra.run.smithi196.stdout:Total 663 kB/s | 145 kB 00:00 2024-03-29T11:19:00.109 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-29T11:19:00.117 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-29T11:19:00.117 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-29T11:19:00.170 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-29T11:19:00.170 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-29T11:19:00.311 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-29T11:19:00.384 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:19:00.386 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:00.409 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:19:00.409 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:19:00.409 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-29T11:19:00.409 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:00.654 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:00.654 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:00.654 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-29T11:19:00.654 INFO:teuthology.orchestra.run.smithi086.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:19:00.655 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:00.655 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T11:19:00.832 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librados2 2024-03-29T11:19:01.377 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d. 1/1 2024-03-29T11:19:01.377 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:01.377 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-29T11:19:01.377 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-immutable-object-cache-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:19:01.377 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:01.377 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T11:19:01.395 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:20 ago on Fri 29 Mar 2024 11:16:41 AM UTC. 2024-03-29T11:19:01.492 INFO:teuthology.orchestra.run.smithi086.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:01.543 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T11:19:01.544 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-29T11:19:01.544 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T11:19:01.595 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install librbd1 2024-03-29T11:19:01.636 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr 2024-03-29T11:19:02.156 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:21 ago on Fri 29 Mar 2024 11:16:41 AM UTC. 2024-03-29T11:19:02.213 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:02:23 ago on Fri 29 Mar 2024 11:16:39 AM UTC. 2024-03-29T11:19:02.253 INFO:teuthology.orchestra.run.smithi086.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:02.304 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T11:19:02.305 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-29T11:19:02.305 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T11:19:02.311 INFO:teuthology.orchestra.run.smithi196.stdout:Package ceph-mgr-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:02.356 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rados 2024-03-29T11:19:02.364 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T11:19:02.365 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-29T11:19:02.365 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T11:19:02.436 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-dashboard 2024-03-29T11:19:02.920 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:21 ago on Fri 29 Mar 2024 11:16:41 AM UTC. 2024-03-29T11:19:02.998 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:02:24 ago on Fri 29 Mar 2024 11:16:39 AM UTC. 2024-03-29T11:19:03.019 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:03.071 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T11:19:03.072 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-29T11:19:03.072 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T11:19:03.132 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rgw 2024-03-29T11:19:03.149 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T11:19:03.150 INFO:teuthology.orchestra.run.smithi196.stdout:====================================================================================== 2024-03-29T11:19:03.150 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-29T11:19:03.150 INFO:teuthology.orchestra.run.smithi196.stdout:====================================================================================== 2024-03-29T11:19:03.150 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-29T11:19:03.150 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 1.8 M 2024-03-29T11:19:03.150 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-29T11:19:03.151 INFO:teuthology.orchestra.run.smithi196.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-29T11:19:03.151 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 31 k 2024-03-29T11:19:03.151 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 15 k 2024-03-29T11:19:03.151 INFO:teuthology.orchestra.run.smithi196.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-29T11:19:03.151 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-29T11:19:03.151 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-29T11:19:03.151 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-29T11:19:03.151 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-29T11:19:03.151 INFO:teuthology.orchestra.run.smithi196.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-29T11:19:03.151 INFO:teuthology.orchestra.run.smithi196.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-29T11:19:03.151 INFO:teuthology.orchestra.run.smithi196.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-29T11:19:03.151 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:03.151 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-29T11:19:03.151 INFO:teuthology.orchestra.run.smithi196.stdout:====================================================================================== 2024-03-29T11:19:03.151 INFO:teuthology.orchestra.run.smithi196.stdout:Install 12 Packages 2024-03-29T11:19:03.152 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:03.152 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 6.9 M 2024-03-29T11:19:03.152 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 26 M 2024-03-29T11:19:03.152 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-29T11:19:03.462 INFO:teuthology.orchestra.run.smithi196.stdout:(1/12): ceph-prometheus-alerts-19.0.0-1439.g5dc 78 kB/s | 15 kB 00:00 2024-03-29T11:19:03.520 INFO:teuthology.orchestra.run.smithi196.stdout:(2/12): ceph-grafana-dashboards-19.0.0-1439.g5d 123 kB/s | 31 kB 00:00 2024-03-29T11:19:03.579 INFO:teuthology.orchestra.run.smithi196.stdout:(3/12): python3-protobuf-3.14.0-13.el9.noarch.r 4.1 MB/s | 238 kB 00:00 2024-03-29T11:19:03.621 INFO:teuthology.orchestra.run.smithi196.stdout:(4/12): protobuf-3.14.0-13.el9.x86_64.rpm 6.3 MB/s | 1.0 MB 00:00 2024-03-29T11:19:03.680 INFO:teuthology.orchestra.run.smithi196.stdout:(5/12): ceph-mgr-dashboard-19.0.0-1439.g5dcb190 4.3 MB/s | 1.8 MB 00:00 2024-03-29T11:19:03.693 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:22 ago on Fri 29 Mar 2024 11:16:41 AM UTC. 2024-03-29T11:19:03.791 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:03.830 INFO:teuthology.orchestra.run.smithi196.stdout:(6/12): protobuf-compiler-3.14.0-13.el9.x86_64. 3.4 MB/s | 863 kB 00:00 2024-03-29T11:19:03.843 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T11:19:03.844 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-29T11:19:03.844 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T11:19:03.896 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-cephfs 2024-03-29T11:19:03.955 INFO:teuthology.orchestra.run.smithi196.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 71 kB/s | 19 kB 00:00 2024-03-29T11:19:04.222 INFO:teuthology.orchestra.run.smithi196.stdout:(8/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 912 kB/s | 548 kB 00:00 2024-03-29T11:19:04.462 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:23 ago on Fri 29 Mar 2024 11:16:41 AM UTC. 2024-03-29T11:19:04.464 INFO:teuthology.orchestra.run.smithi196.stdout:(9/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 284 kB/s | 144 kB 00:00 2024-03-29T11:19:04.489 INFO:teuthology.orchestra.run.smithi196.stdout:(10/12): python3-repoze-lru-0.7-16.el9.noarch.r 115 kB/s | 31 kB 00:00 2024-03-29T11:19:04.559 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:04.610 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T11:19:04.611 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-29T11:19:04.611 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T11:19:04.673 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install python3-rbd 2024-03-29T11:19:04.689 INFO:teuthology.orchestra.run.smithi196.stdout:(11/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 2.4 MB/s | 2.0 MB 00:00 2024-03-29T11:19:04.890 INFO:teuthology.orchestra.run.smithi196.stdout:(12/12): python3-routes-2.5.1-5.el9.noarch.rpm 443 kB/s | 188 kB 00:00 2024-03-29T11:19:04.894 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:04.894 INFO:teuthology.orchestra.run.smithi196.stdout:Total 3.9 MB/s | 6.9 MB 00:01 2024-03-29T11:19:04.915 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-29T11:19:04.934 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-29T11:19:04.934 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-29T11:19:05.109 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-29T11:19:05.110 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-29T11:19:05.234 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:24 ago on Fri 29 Mar 2024 11:16:41 AM UTC. 2024-03-29T11:19:05.333 INFO:teuthology.orchestra.run.smithi086.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:05.384 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T11:19:05.385 INFO:teuthology.orchestra.run.smithi086.stdout:Nothing to do. 2024-03-29T11:19:05.385 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T11:19:05.436 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-fuse 2024-03-29T11:19:05.621 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-29T11:19:05.691 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-03-29T11:19:05.726 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-03-29T11:19:05.772 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-03-29T11:19:05.803 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-03-29T11:19:05.906 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-03-29T11:19:05.992 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:25 ago on Fri 29 Mar 2024 11:16:41 AM UTC. 2024-03-29T11:19:06.028 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-03-29T11:19:06.131 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-03-29T11:19:06.140 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T11:19:06.141 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:19:06.141 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-03-29T11:19:06.141 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:19:06.141 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-29T11:19:06.141 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:19:06.141 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:06.141 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-29T11:19:06.141 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:19:06.141 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-29T11:19:06.141 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:06.142 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 85 k 2024-03-29T11:19:06.142 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 231 k 2024-03-29T11:19:06.142 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-29T11:19:06.174 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-03-29T11:19:06.203 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:19:06.234 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 10/12 2024-03-29T11:19:06.479 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 11/12 2024-03-29T11:19:06.518 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:19:06.648 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 168 kB/s | 85 kB 00:00 2024-03-29T11:19:06.648 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:06.649 INFO:teuthology.orchestra.run.smithi086.stdout:Total 168 kB/s | 85 kB 00:00 2024-03-29T11:19:06.649 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-29T11:19:06.656 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-29T11:19:06.656 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-29T11:19:06.711 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-29T11:19:06.712 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-29T11:19:06.849 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-29T11:19:06.955 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:07.294 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 12/12 2024-03-29T11:19:07.294 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el 1/12 2024-03-29T11:19:07.294 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noa 2/12 2024-03-29T11:19:07.294 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9 3/12 2024-03-29T11:19:07.294 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-03-29T11:19:07.294 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-03-29T11:19:07.294 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-03-29T11:19:07.294 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-03-29T11:19:07.294 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-03-29T11:19:07.294 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-03-29T11:19:07.294 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-03-29T11:19:07.295 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-03-29T11:19:07.383 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:07.631 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-03-29T11:19:07.631 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:07.631 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-29T11:19:07.631 INFO:teuthology.orchestra.run.smithi196.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-29T11:19:07.631 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-grafana-dashboards-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:19:07.631 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-dashboard-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:19:07.631 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-prometheus-alerts-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:19:07.632 INFO:teuthology.orchestra.run.smithi196.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-29T11:19:07.632 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-29T11:19:07.632 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-29T11:19:07.632 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-29T11:19:07.632 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-29T11:19:07.632 INFO:teuthology.orchestra.run.smithi196.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-29T11:19:07.632 INFO:teuthology.orchestra.run.smithi196.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-29T11:19:07.632 INFO:teuthology.orchestra.run.smithi196.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-29T11:19:07.632 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:07.632 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T11:19:07.672 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:07.672 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:07.672 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-29T11:19:07.672 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:19:07.672 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:07.672 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T11:19:07.884 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-mirror 2024-03-29T11:19:07.912 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-29T11:19:08.459 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:27 ago on Fri 29 Mar 2024 11:16:41 AM UTC. 2024-03-29T11:19:08.474 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:02:29 ago on Fri 29 Mar 2024 11:16:39 AM UTC. 2024-03-29T11:19:08.614 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T11:19:08.615 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:19:08.615 INFO:teuthology.orchestra.run.smithi086.stdout: Package Arch Version Repo Size 2024-03-29T11:19:08.615 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:19:08.615 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-29T11:19:08.615 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:19:08.615 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:08.615 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-29T11:19:08.615 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:19:08.616 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-29T11:19:08.616 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:08.616 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 3.1 M 2024-03-29T11:19:08.616 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 13 M 2024-03-29T11:19:08.616 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-29T11:19:08.627 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T11:19:08.628 INFO:teuthology.orchestra.run.smithi196.stdout:============================================================================================ 2024-03-29T11:19:08.628 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-29T11:19:08.629 INFO:teuthology.orchestra.run.smithi196.stdout:============================================================================================ 2024-03-29T11:19:08.629 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-29T11:19:08.629 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 7.4 M 2024-03-29T11:19:08.629 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-29T11:19:08.629 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-29T11:19:08.629 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-29T11:19:08.629 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-29T11:19:08.629 INFO:teuthology.orchestra.run.smithi196.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-29T11:19:08.629 INFO:teuthology.orchestra.run.smithi196.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-29T11:19:08.629 INFO:teuthology.orchestra.run.smithi196.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-29T11:19:08.629 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-29T11:19:08.629 INFO:teuthology.orchestra.run.smithi196.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-29T11:19:08.629 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-29T11:19:08.629 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-29T11:19:08.629 INFO:teuthology.orchestra.run.smithi196.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-29T11:19:08.630 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:08.630 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-29T11:19:08.630 INFO:teuthology.orchestra.run.smithi196.stdout:============================================================================================ 2024-03-29T11:19:08.630 INFO:teuthology.orchestra.run.smithi196.stdout:Install 12 Packages 2024-03-29T11:19:08.630 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:08.630 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 37 M 2024-03-29T11:19:08.630 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 209 M 2024-03-29T11:19:08.631 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-29T11:19:09.180 INFO:teuthology.orchestra.run.smithi196.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 220 kB/s | 32 kB 00:00 2024-03-29T11:19:09.214 INFO:teuthology.orchestra.run.smithi196.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 532 kB/s | 17 kB 00:00 2024-03-29T11:19:09.264 INFO:teuthology.orchestra.run.smithi196.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 698 kB/s | 34 kB 00:00 2024-03-29T11:19:09.322 INFO:teuthology.orchestra.run.smithi196.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 10 MB/s | 3.0 MB 00:00 2024-03-29T11:19:09.531 INFO:teuthology.orchestra.run.smithi196.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 18 MB/s | 4.7 MB 00:00 2024-03-29T11:19:09.560 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 3.3 MB/s | 3.1 MB 00:00 2024-03-29T11:19:09.561 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:09.561 INFO:teuthology.orchestra.run.smithi086.stdout:Total 3.3 MB/s | 3.1 MB 00:00 2024-03-29T11:19:09.561 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-29T11:19:09.571 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-29T11:19:09.571 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-29T11:19:09.598 INFO:teuthology.orchestra.run.smithi196.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 17 MB/s | 4.8 MB 00:00 2024-03-29T11:19:09.623 INFO:teuthology.orchestra.run.smithi196.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.4 MB/s | 217 kB 00:00 2024-03-29T11:19:09.644 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-29T11:19:09.644 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-29T11:19:09.815 INFO:teuthology.orchestra.run.smithi196.stdout:(8/12): libgfortran-11.4.1-3.el9.x86_64.rpm 4.1 MB/s | 803 kB 00:00 2024-03-29T11:19:09.966 INFO:teuthology.orchestra.run.smithi196.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 43 MB/s | 16 MB 00:00 2024-03-29T11:19:09.991 INFO:teuthology.orchestra.run.smithi196.stdout:(10/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-03-29T11:19:10.081 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-29T11:19:10.136 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:10.160 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:10.160 INFO:teuthology.orchestra.run.smithi086.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:19:10.160 INFO:teuthology.orchestra.run.smithi086.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:19:10.160 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:19:10.160 INFO:teuthology.orchestra.run.smithi086.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:19:10.160 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:10.166 INFO:teuthology.orchestra.run.smithi196.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.2 MB/s | 245 kB 00:00 2024-03-29T11:19:10.508 INFO:teuthology.orchestra.run.smithi196.stdout:(12/12): ceph-mgr-diskprediction-local-19.0.0-1 5.0 MB/s | 7.4 MB 00:01 2024-03-29T11:19:10.510 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:10.511 INFO:teuthology.orchestra.run.smithi196.stdout:Total 20 MB/s | 37 MB 00:01 2024-03-29T11:19:10.552 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-29T11:19:10.576 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-29T11:19:10.577 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-29T11:19:10.933 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-29T11:19:10.934 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-29T11:19:11.306 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:11.306 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:11.306 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-29T11:19:11.306 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:19:11.306 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:11.306 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T11:19:11.574 DEBUG:teuthology.orchestra.run.smithi086:> sudo yum -y install rbd-nbd 2024-03-29T11:19:11.638 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-29T11:19:11.697 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-29T11:19:11.729 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-29T11:19:11.822 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:19:11.865 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-29T11:19:12.045 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:19:12.081 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:19:12.146 INFO:teuthology.orchestra.run.smithi086.stdout:Last metadata expiration check: 0:02:31 ago on Fri 29 Mar 2024 11:16:41 AM UTC. 2024-03-29T11:19:12.163 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-29T11:19:12.294 INFO:teuthology.orchestra.run.smithi086.stdout:Dependencies resolved. 2024-03-29T11:19:12.295 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:19:12.295 INFO:teuthology.orchestra.run.smithi086.stdout: Package Architecture Version Repository Size 2024-03-29T11:19:12.295 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:19:12.295 INFO:teuthology.orchestra.run.smithi086.stdout:Installing: 2024-03-29T11:19:12.295 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:19:12.295 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:12.296 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction Summary 2024-03-29T11:19:12.296 INFO:teuthology.orchestra.run.smithi086.stdout:================================================================================ 2024-03-29T11:19:12.296 INFO:teuthology.orchestra.run.smithi086.stdout:Install 1 Package 2024-03-29T11:19:12.296 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:12.296 INFO:teuthology.orchestra.run.smithi086.stdout:Total download size: 171 k 2024-03-29T11:19:12.296 INFO:teuthology.orchestra.run.smithi086.stdout:Installed size: 494 k 2024-03-29T11:19:12.296 INFO:teuthology.orchestra.run.smithi086.stdout:Downloading Packages: 2024-03-29T11:19:12.613 INFO:teuthology.orchestra.run.smithi086.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 542 kB/s | 171 kB 00:00 2024-03-29T11:19:12.614 INFO:teuthology.orchestra.run.smithi086.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:12.614 INFO:teuthology.orchestra.run.smithi086.stdout:Total 539 kB/s | 171 kB 00:00 2024-03-29T11:19:12.614 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction check 2024-03-29T11:19:12.622 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction check succeeded. 2024-03-29T11:19:12.622 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction test 2024-03-29T11:19:12.677 INFO:teuthology.orchestra.run.smithi086.stdout:Transaction test succeeded. 2024-03-29T11:19:12.678 INFO:teuthology.orchestra.run.smithi086.stdout:Running transaction 2024-03-29T11:19:12.811 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-29T11:19:12.830 INFO:teuthology.orchestra.run.smithi086.stdout: Preparing : 1/1 2024-03-29T11:19:12.926 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-29T11:19:12.945 INFO:teuthology.orchestra.run.smithi086.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:13.372 INFO:teuthology.orchestra.run.smithi086.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:13.671 INFO:teuthology.orchestra.run.smithi086.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:13.671 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:13.671 INFO:teuthology.orchestra.run.smithi086.stdout:Installed: 2024-03-29T11:19:13.671 INFO:teuthology.orchestra.run.smithi086.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:19:13.671 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:13.672 INFO:teuthology.orchestra.run.smithi086.stdout:Complete! 2024-03-29T11:19:13.949 DEBUG:teuthology.parallel:result is None 2024-03-29T11:19:14.154 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-29T11:19:15.190 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-29T11:19:15.214 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:19:17.699 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 12/12 2024-03-29T11:19:17.699 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb1 1/12 2024-03-29T11:19:17.699 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-29T11:19:17.699 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-29T11:19:17.699 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-29T11:19:17.699 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-29T11:19:17.699 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-29T11:19:17.699 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-29T11:19:17.700 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-29T11:19:17.700 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-29T11:19:17.700 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-29T11:19:17.700 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-29T11:19:18.136 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-29T11:19:18.136 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:18.136 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-29T11:19:18.136 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:19:18.136 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-29T11:19:18.136 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-29T11:19:18.136 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-29T11:19:18.136 INFO:teuthology.orchestra.run.smithi196.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-29T11:19:18.136 INFO:teuthology.orchestra.run.smithi196.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-29T11:19:18.136 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-29T11:19:18.136 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-29T11:19:18.137 INFO:teuthology.orchestra.run.smithi196.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-29T11:19:18.137 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-29T11:19:18.137 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-29T11:19:18.137 INFO:teuthology.orchestra.run.smithi196.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-29T11:19:18.137 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:18.137 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T11:19:18.501 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-rook 2024-03-29T11:19:19.073 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:02:40 ago on Fri 29 Mar 2024 11:16:39 AM UTC. 2024-03-29T11:19:19.224 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T11:19:19.225 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-03-29T11:19:19.225 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-29T11:19:19.225 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-03-29T11:19:19.225 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-29T11:19:19.226 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-rook noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 52 k 2024-03-29T11:19:19.226 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-29T11:19:19.226 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-29T11:19:19.226 INFO:teuthology.orchestra.run.smithi196.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-29T11:19:19.226 INFO:teuthology.orchestra.run.smithi196.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-29T11:19:19.226 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-29T11:19:19.226 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-29T11:19:19.226 INFO:teuthology.orchestra.run.smithi196.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-29T11:19:19.226 INFO:teuthology.orchestra.run.smithi196.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-29T11:19:19.226 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-29T11:19:19.226 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-29T11:19:19.226 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-29T11:19:19.226 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-29T11:19:19.227 INFO:teuthology.orchestra.run.smithi196.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-29T11:19:19.227 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:19.227 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-29T11:19:19.227 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-03-29T11:19:19.227 INFO:teuthology.orchestra.run.smithi196.stdout:Install 13 Packages 2024-03-29T11:19:19.227 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:19.227 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 2.0 M 2024-03-29T11:19:19.227 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 26 M 2024-03-29T11:19:19.228 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-29T11:19:19.431 INFO:teuthology.orchestra.run.smithi196.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 193 kB/s | 24 kB 00:00 2024-03-29T11:19:19.456 INFO:teuthology.orchestra.run.smithi196.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 119 kB/s | 18 kB 00:00 2024-03-29T11:19:19.498 INFO:teuthology.orchestra.run.smithi196.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.6 MB/s | 174 kB 00:00 2024-03-29T11:19:19.523 INFO:teuthology.orchestra.run.smithi196.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-29T11:19:19.548 INFO:teuthology.orchestra.run.smithi196.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-29T11:19:19.574 INFO:teuthology.orchestra.run.smithi196.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 853 kB/s | 43 kB 00:00 2024-03-29T11:19:19.599 INFO:teuthology.orchestra.run.smithi196.stdout:(7/13): ceph-mgr-rook-19.0.0-1439.g5dcb190d.el9 177 kB/s | 52 kB 00:00 2024-03-29T11:19:19.691 INFO:teuthology.orchestra.run.smithi196.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 128 kB/s | 15 kB 00:00 2024-03-29T11:19:19.708 INFO:teuthology.orchestra.run.smithi196.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 202 kB/s | 32 kB 00:00 2024-03-29T11:19:19.816 INFO:teuthology.orchestra.run.smithi196.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 545 kB/s | 59 kB 00:00 2024-03-29T11:19:19.842 INFO:teuthology.orchestra.run.smithi196.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 836 kB/s | 203 kB 00:00 2024-03-29T11:19:19.900 INFO:teuthology.orchestra.run.smithi196.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.0 MB/s | 90 kB 00:00 2024-03-29T11:19:19.959 INFO:teuthology.orchestra.run.smithi196.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.8 MB/s | 1.0 MB 00:00 2024-03-29T11:19:19.960 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:19.961 INFO:teuthology.orchestra.run.smithi196.stdout:Total 2.8 MB/s | 2.0 MB 00:00 2024-03-29T11:19:19.961 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-29T11:19:19.974 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-29T11:19:19.974 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-29T11:19:20.064 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-29T11:19:20.065 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-29T11:19:20.579 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-29T11:19:20.754 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-29T11:19:20.801 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-29T11:19:20.848 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-29T11:19:20.877 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-29T11:19:20.918 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-29T11:19:21.011 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-29T11:19:21.120 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-29T11:19:21.170 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-29T11:19:21.877 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-29T11:19:21.934 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-29T11:19:21.976 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-29T11:19:22.024 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-29T11:19:22.046 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:19:22.862 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 13/13 2024-03-29T11:19:22.862 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 1/13 2024-03-29T11:19:22.862 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-29T11:19:22.862 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-29T11:19:22.862 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-29T11:19:22.863 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-29T11:19:22.863 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-29T11:19:22.863 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-29T11:19:22.863 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-29T11:19:22.863 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-29T11:19:22.863 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-29T11:19:22.863 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-29T11:19:22.863 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-29T11:19:23.224 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-29T11:19:23.224 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:23.224 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-29T11:19:23.225 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-rook-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:19:23.225 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-29T11:19:23.225 INFO:teuthology.orchestra.run.smithi196.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-29T11:19:23.225 INFO:teuthology.orchestra.run.smithi196.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-29T11:19:23.225 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-29T11:19:23.225 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-29T11:19:23.225 INFO:teuthology.orchestra.run.smithi196.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-29T11:19:23.225 INFO:teuthology.orchestra.run.smithi196.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-29T11:19:23.225 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-29T11:19:23.225 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-29T11:19:23.225 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-29T11:19:23.225 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-29T11:19:23.225 INFO:teuthology.orchestra.run.smithi196.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-29T11:19:23.225 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:23.226 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T11:19:23.467 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-mgr-cephadm 2024-03-29T11:19:24.036 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:02:45 ago on Fri 29 Mar 2024 11:16:39 AM UTC. 2024-03-29T11:19:24.189 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T11:19:24.190 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-03-29T11:19:24.190 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-29T11:19:24.190 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-03-29T11:19:24.190 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-29T11:19:24.190 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 138 k 2024-03-29T11:19:24.190 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-29T11:19:24.190 INFO:teuthology.orchestra.run.smithi196.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-29T11:19:24.190 INFO:teuthology.orchestra.run.smithi196.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-29T11:19:24.190 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-29T11:19:24.190 INFO:teuthology.orchestra.run.smithi196.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-29T11:19:24.191 INFO:teuthology.orchestra.run.smithi196.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-29T11:19:24.191 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:24.191 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-29T11:19:24.191 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================== 2024-03-29T11:19:24.191 INFO:teuthology.orchestra.run.smithi196.stdout:Install 6 Packages 2024-03-29T11:19:24.191 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:24.191 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 6.8 M 2024-03-29T11:19:24.192 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 33 M 2024-03-29T11:19:24.192 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-29T11:19:24.578 INFO:teuthology.orchestra.run.smithi196.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 23 MB/s | 5.8 MB 00:00 2024-03-29T11:19:24.629 INFO:teuthology.orchestra.run.smithi196.stdout:(2/6): ceph-mgr-cephadm-19.0.0-1439.g5dcb190d.e 463 kB/s | 138 kB 00:00 2024-03-29T11:19:24.979 INFO:teuthology.orchestra.run.smithi196.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 165 kB/s | 58 kB 00:00 2024-03-29T11:19:25.229 INFO:teuthology.orchestra.run.smithi196.stdout:(4/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 874 kB/s | 548 kB 00:00 2024-03-29T11:19:25.323 INFO:teuthology.orchestra.run.smithi196.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 148 kB/s | 51 kB 00:00 2024-03-29T11:19:25.348 INFO:teuthology.orchestra.run.smithi196.stdout:(6/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 244 kB/s | 248 kB 00:01 2024-03-29T11:19:25.350 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:25.351 INFO:teuthology.orchestra.run.smithi196.stdout:Total 5.9 MB/s | 6.8 MB 00:01 2024-03-29T11:19:25.364 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-29T11:19:25.375 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-29T11:19:25.375 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-29T11:19:25.463 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-29T11:19:25.464 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-29T11:19:25.758 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-29T11:19:25.927 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-29T11:19:25.982 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-29T11:19:26.490 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-29T11:19:26.568 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-29T11:19:26.630 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-29T11:19:26.653 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:19:27.482 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 6/6 2024-03-29T11:19:27.483 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 1/6 2024-03-29T11:19:27.483 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-29T11:19:27.483 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-29T11:19:27.483 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-29T11:19:27.483 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-29T11:19:27.778 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-29T11:19:27.779 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:27.779 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-29T11:19:27.779 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-cephadm-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:19:27.779 INFO:teuthology.orchestra.run.smithi196.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-29T11:19:27.779 INFO:teuthology.orchestra.run.smithi196.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-29T11:19:27.779 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-29T11:19:27.779 INFO:teuthology.orchestra.run.smithi196.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-29T11:19:27.779 INFO:teuthology.orchestra.run.smithi196.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-29T11:19:27.779 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:27.779 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T11:19:28.020 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-fuse 2024-03-29T11:19:28.626 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:02:49 ago on Fri 29 Mar 2024 11:16:39 AM UTC. 2024-03-29T11:19:28.777 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T11:19:28.777 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:19:28.777 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-29T11:19:28.777 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:19:28.778 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-29T11:19:28.778 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 885 k 2024-03-29T11:19:28.778 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-29T11:19:28.778 INFO:teuthology.orchestra.run.smithi196.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-29T11:19:28.778 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:28.778 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-29T11:19:28.778 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:19:28.778 INFO:teuthology.orchestra.run.smithi196.stdout:Install 2 Packages 2024-03-29T11:19:28.778 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:28.779 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 964 k 2024-03-29T11:19:28.779 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 2.8 M 2024-03-29T11:19:28.779 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-29T11:19:28.917 INFO:teuthology.orchestra.run.smithi196.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 571 kB/s | 79 kB 00:00 2024-03-29T11:19:29.267 INFO:teuthology.orchestra.run.smithi196.stdout:(2/2): ceph-fuse-19.0.0-1439.g5dcb190d.el9.x86_ 1.8 MB/s | 885 kB 00:00 2024-03-29T11:19:29.268 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:29.268 INFO:teuthology.orchestra.run.smithi196.stdout:Total 1.9 MB/s | 964 kB 00:00 2024-03-29T11:19:29.268 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-29T11:19:29.289 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-29T11:19:29.289 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-29T11:19:29.363 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-29T11:19:29.363 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-29T11:19:29.544 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-29T11:19:29.684 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-29T11:19:29.759 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:19:30.457 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2/2 2024-03-29T11:19:30.458 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/2 2024-03-29T11:19:30.762 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-29T11:19:30.762 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:30.763 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-29T11:19:30.763 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-29T11:19:30.763 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:30.763 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T11:19:31.054 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-volume 2024-03-29T11:19:31.631 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:02:52 ago on Fri 29 Mar 2024 11:16:39 AM UTC. 2024-03-29T11:19:31.783 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T11:19:31.784 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:19:31.784 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-03-29T11:19:31.784 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:19:31.784 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-29T11:19:31.784 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-volume noarch 2:19.0.0-1439.g5dcb190d.el9 ceph-noarch 264 k 2024-03-29T11:19:31.784 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-03-29T11:19:31.784 INFO:teuthology.orchestra.run.smithi196.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-29T11:19:31.784 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:31.784 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-29T11:19:31.784 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:19:31.784 INFO:teuthology.orchestra.run.smithi196.stdout:Install 2 Packages 2024-03-29T11:19:31.784 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:31.785 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 332 k 2024-03-29T11:19:31.785 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 1.5 M 2024-03-29T11:19:31.785 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-29T11:19:31.905 INFO:teuthology.orchestra.run.smithi196.stdout:(1/2): python3-packaging-20.9-5.el9.noarch.rpm 579 kB/s | 69 kB 00:00 2024-03-29T11:19:32.188 INFO:teuthology.orchestra.run.smithi196.stdout:(2/2): ceph-volume-19.0.0-1439.g5dcb190d.el9.no 656 kB/s | 264 kB 00:00 2024-03-29T11:19:32.189 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:32.190 INFO:teuthology.orchestra.run.smithi196.stdout:Total 821 kB/s | 332 kB 00:00 2024-03-29T11:19:32.190 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-29T11:19:32.198 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-29T11:19:32.198 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-29T11:19:32.242 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-29T11:19:32.243 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-29T11:19:32.393 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-29T11:19:32.580 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-packaging-20.9-5.el9.noarch 1/2 2024-03-29T11:19:32.616 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:19:32.634 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2/2 2024-03-29T11:19:32.634 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:19:32.634 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-29T11:19:32.634 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:33.307 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 1/2 2024-03-29T11:19:33.650 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 2/2 2024-03-29T11:19:33.650 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:33.650 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-29T11:19:33.650 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-volume-2:19.0.0-1439.g5dcb190d.el9.noarch 2024-03-29T11:19:33.650 INFO:teuthology.orchestra.run.smithi196.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-29T11:19:33.650 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:33.650 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T11:19:34.004 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librados-devel 2024-03-29T11:19:34.586 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:02:55 ago on Fri 29 Mar 2024 11:16:39 AM UTC. 2024-03-29T11:19:34.740 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T11:19:34.741 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:19:34.741 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-29T11:19:34.741 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:19:34.741 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-29T11:19:34.741 INFO:teuthology.orchestra.run.smithi196.stdout: librados-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 127 k 2024-03-29T11:19:34.741 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:34.741 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-29T11:19:34.741 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:19:34.741 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-29T11:19:34.741 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:34.741 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 127 k 2024-03-29T11:19:34.742 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 456 k 2024-03-29T11:19:34.742 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-29T11:19:34.944 INFO:teuthology.orchestra.run.smithi196.stdout:librados-devel-19.0.0-1439.g5dcb190d.el9.x86_64 627 kB/s | 127 kB 00:00 2024-03-29T11:19:34.944 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:34.945 INFO:teuthology.orchestra.run.smithi196.stdout:Total 622 kB/s | 127 kB 00:00 2024-03-29T11:19:34.945 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-29T11:19:34.950 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-29T11:19:34.950 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-29T11:19:35.006 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-29T11:19:35.006 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-29T11:19:35.146 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-29T11:19:35.250 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:35.712 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:36.008 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:36.009 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:36.009 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-29T11:19:36.009 INFO:teuthology.orchestra.run.smithi196.stdout: librados-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:19:36.009 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:36.009 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T11:19:36.228 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install libcephfs2 2024-03-29T11:19:36.800 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:02:57 ago on Fri 29 Mar 2024 11:16:39 AM UTC. 2024-03-29T11:19:36.901 INFO:teuthology.orchestra.run.smithi196.stdout:Package libcephfs2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:36.957 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T11:19:36.958 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-29T11:19:36.958 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T11:19:37.026 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install libcephfs-devel 2024-03-29T11:19:37.590 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:02:58 ago on Fri 29 Mar 2024 11:16:39 AM UTC. 2024-03-29T11:19:37.743 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T11:19:37.744 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:19:37.744 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-29T11:19:37.744 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:19:37.744 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-29T11:19:37.744 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs-devel x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 31 k 2024-03-29T11:19:37.744 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:37.744 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-29T11:19:37.744 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:19:37.744 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-29T11:19:37.744 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:37.744 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 31 k 2024-03-29T11:19:37.744 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 140 k 2024-03-29T11:19:37.744 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-29T11:19:37.914 INFO:teuthology.orchestra.run.smithi196.stdout:libcephfs-devel-19.0.0-1439.g5dcb190d.el9.x86_6 184 kB/s | 31 kB 00:00 2024-03-29T11:19:37.915 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:37.915 INFO:teuthology.orchestra.run.smithi196.stdout:Total 182 kB/s | 31 kB 00:00 2024-03-29T11:19:37.915 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-29T11:19:37.918 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-29T11:19:37.919 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-29T11:19:37.955 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-29T11:19:37.956 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-29T11:19:38.084 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-29T11:19:38.169 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:38.555 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:38.825 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:38.825 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:38.825 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-29T11:19:38.826 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs-devel-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:19:38.826 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:38.826 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T11:19:38.970 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librados2 2024-03-29T11:19:39.537 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:03:00 ago on Fri 29 Mar 2024 11:16:39 AM UTC. 2024-03-29T11:19:39.634 INFO:teuthology.orchestra.run.smithi196.stdout:Package librados2-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:39.685 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T11:19:39.686 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-29T11:19:39.686 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T11:19:39.738 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install librbd1 2024-03-29T11:19:40.307 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:03:01 ago on Fri 29 Mar 2024 11:16:39 AM UTC. 2024-03-29T11:19:40.406 INFO:teuthology.orchestra.run.smithi196.stdout:Package librbd1-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:40.458 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T11:19:40.459 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-29T11:19:40.459 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T11:19:40.513 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rados 2024-03-29T11:19:41.084 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:03:02 ago on Fri 29 Mar 2024 11:16:39 AM UTC. 2024-03-29T11:19:41.183 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rados-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:41.234 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T11:19:41.238 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-29T11:19:41.238 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T11:19:41.283 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rgw 2024-03-29T11:19:41.848 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:03:02 ago on Fri 29 Mar 2024 11:16:39 AM UTC. 2024-03-29T11:19:41.948 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rgw-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:41.999 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T11:19:42.000 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-29T11:19:42.000 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T11:19:42.045 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-cephfs 2024-03-29T11:19:42.612 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:03:03 ago on Fri 29 Mar 2024 11:16:39 AM UTC. 2024-03-29T11:19:42.710 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-cephfs-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:42.761 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T11:19:42.762 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-29T11:19:42.762 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T11:19:42.813 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install python3-rbd 2024-03-29T11:19:43.380 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:03:04 ago on Fri 29 Mar 2024 11:16:39 AM UTC. 2024-03-29T11:19:43.479 INFO:teuthology.orchestra.run.smithi196.stdout:Package python3-rbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 is already installed. 2024-03-29T11:19:43.530 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T11:19:43.531 INFO:teuthology.orchestra.run.smithi196.stdout:Nothing to do. 2024-03-29T11:19:43.532 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T11:19:43.575 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-fuse 2024-03-29T11:19:44.134 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:03:05 ago on Fri 29 Mar 2024 11:16:39 AM UTC. 2024-03-29T11:19:44.281 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T11:19:44.282 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:19:44.282 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-03-29T11:19:44.282 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:19:44.282 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-29T11:19:44.282 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-fuse x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 85 k 2024-03-29T11:19:44.282 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:44.282 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-29T11:19:44.282 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:19:44.283 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-29T11:19:44.283 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:44.283 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 85 k 2024-03-29T11:19:44.283 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 231 k 2024-03-29T11:19:44.283 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-29T11:19:44.592 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-fuse-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 276 kB/s | 85 kB 00:00 2024-03-29T11:19:44.592 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:44.592 INFO:teuthology.orchestra.run.smithi196.stdout:Total 275 kB/s | 85 kB 00:00 2024-03-29T11:19:44.593 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-29T11:19:44.599 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-29T11:19:44.599 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-29T11:19:44.654 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-29T11:19:44.655 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-29T11:19:44.815 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-29T11:19:44.923 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:45.303 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:45.632 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:45.632 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:45.632 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-29T11:19:45.632 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-fuse-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:19:45.632 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:45.633 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T11:19:45.903 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-mirror 2024-03-29T11:19:46.476 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:03:07 ago on Fri 29 Mar 2024 11:16:39 AM UTC. 2024-03-29T11:19:46.631 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T11:19:46.632 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:19:46.632 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repo Size 2024-03-29T11:19:46.632 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:19:46.632 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-29T11:19:46.632 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-mirror x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 3.1 M 2024-03-29T11:19:46.633 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:46.633 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-29T11:19:46.633 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:19:46.633 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-29T11:19:46.633 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:46.633 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 3.1 M 2024-03-29T11:19:46.633 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 13 M 2024-03-29T11:19:46.633 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-29T11:19:47.703 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-mirror-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 2.9 MB/s | 3.1 MB 00:01 2024-03-29T11:19:47.704 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:47.704 INFO:teuthology.orchestra.run.smithi196.stdout:Total 2.9 MB/s | 3.1 MB 00:01 2024-03-29T11:19:47.704 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-29T11:19:47.715 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-29T11:19:47.715 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-29T11:19:47.792 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-29T11:19:47.792 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-29T11:19:48.225 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-29T11:19:48.303 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:48.328 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:48.328 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-29T11:19:48.328 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-29T11:19:48.328 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:19:48.329 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-29T11:19:48.329 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:49.445 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:49.445 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:49.445 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-29T11:19:49.445 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-mirror-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:19:49.445 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:49.445 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T11:19:49.689 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install rbd-nbd 2024-03-29T11:19:50.265 INFO:teuthology.orchestra.run.smithi196.stdout:Last metadata expiration check: 0:03:11 ago on Fri 29 Mar 2024 11:16:39 AM UTC. 2024-03-29T11:19:50.416 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-03-29T11:19:50.417 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:19:50.417 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-03-29T11:19:50.417 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:19:50.417 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-03-29T11:19:50.417 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-nbd x86_64 2:19.0.0-1439.g5dcb190d.el9 ceph 171 k 2024-03-29T11:19:50.417 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:50.417 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-03-29T11:19:50.417 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-03-29T11:19:50.417 INFO:teuthology.orchestra.run.smithi196.stdout:Install 1 Package 2024-03-29T11:19:50.417 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:50.418 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 171 k 2024-03-29T11:19:50.418 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 494 k 2024-03-29T11:19:50.418 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-03-29T11:19:50.631 INFO:teuthology.orchestra.run.smithi196.stdout:rbd-nbd-19.0.0-1439.g5dcb190d.el9.x86_64.rpm 803 kB/s | 171 kB 00:00 2024-03-29T11:19:50.632 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-03-29T11:19:50.632 INFO:teuthology.orchestra.run.smithi196.stdout:Total 798 kB/s | 171 kB 00:00 2024-03-29T11:19:50.632 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-03-29T11:19:50.640 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-03-29T11:19:50.640 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-03-29T11:19:50.696 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-03-29T11:19:50.696 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-03-29T11:19:50.852 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-03-29T11:19:50.955 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:51.454 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:51.748 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 1/1 2024-03-29T11:19:51.748 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:51.748 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-03-29T11:19:51.748 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-nbd-2:19.0.0-1439.g5dcb190d.el9.x86_64 2024-03-29T11:19:51.748 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:51.748 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-03-29T11:19:51.967 DEBUG:teuthology.parallel:result is None 2024-03-29T11:19:51.968 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:19:52.113 DEBUG:teuthology.orchestra.run.smithi086:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:19:52.138 INFO:teuthology.orchestra.run.smithi086.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:19:52.138 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:19:52.138 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:19:52.140 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5dcb190da270440ef141d088eb27a9fdb8afc5f2 2024-03-29T11:19:52.450 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-29T11:19:52.470 INFO:teuthology.orchestra.run.smithi196.stdout:19.0.0-1439.g5dcb190d.el9 2024-03-29T11:19:52.470 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1439.g5dcb190d.el9 2024-03-29T11:19:52.470 INFO:teuthology.task.install:The correct ceph version 19.0.0-1439.g5dcb190d is installed. 2024-03-29T11:19:52.472 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-29T11:19:52.472 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T11:19:52.472 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:19:52.505 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T11:19:52.505 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-29T11:19:52.538 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-29T11:19:52.538 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T11:19:52.538 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:19:52.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:19:52.646 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T11:19:52.646 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/daemon-helper 2024-03-29T11:19:52.676 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-29T11:19:52.743 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-29T11:19:52.744 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T11:19:52.744 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:19:52.776 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:19:52.845 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T11:19:52.845 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-29T11:19:52.871 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-29T11:19:52.938 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-29T11:19:52.939 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T11:19:52.939 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:19:52.966 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:19:53.033 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T11:19:53.033 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/stdin-killer 2024-03-29T11:19:53.059 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-29T11:19:53.127 INFO:teuthology.run_tasks:Running task ceph... 2024-03-29T11:19:53.209 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-03-29T11:19:53.209 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 777 /var/log/ceph 2024-03-29T11:19:53.211 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 777 /var/log/ceph 2024-03-29T11:19:53.238 INFO:tasks.ceph:Disabling ceph logrotate... 2024-03-29T11:19:53.238 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:19:53.280 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /etc/logrotate.d/ceph 2024-03-29T11:19:53.306 INFO:tasks.ceph:Creating extra log directories... 2024-03-29T11:19:53.307 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:19:53.349 DEBUG:teuthology.orchestra.run.smithi196:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-03-29T11:19:53.386 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-03-29T11:19:53.386 INFO:tasks.ceph:config {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2024-03-29T11:19:53.386 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622694', 'branch': 'main', 'description': 'rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep}', 'email': None, 'first_in_suite': False, 'job_id': '7622694', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 10}}, {'volumes': {'count': 4, 'size': 10}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'main'}, 'ceph': {'conf': {'client': {'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'global': {'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd mclock profile': 'high_recovery_ops', 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME', 'reached quota', 'overall HEALTH_', '\\(CACHE_POOL_NO_HIT_SET\\)', '\\(POOL_FULL\\)', '\\(SMALLER_PGP_NUM\\)', '\\(SLOW_OPS\\)', '\\(CACHE_POOL_NEAR_FULL\\)', '\\(POOL_APP_NOT_ENABLED\\)', '\\(PG_AVAILABILITY\\)', '\\(OBJECT_MISPLACED\\)', 'slow request'], 'mon_bind_msgr2': False, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'main', 'sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324'}}, 'owner': 'scheduled_amathuri@teuthology', 'priority': 150, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'node-exporter.a'], ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'node-exporter.b']], 'seed': 8152, 'sha1': '5dcb190da270440ef141d088eb27a9fdb8afc5f2', 'sleep_before_teardown': 0, 'subset': '99/1000', 'suite': 'rados:verify', 'suite_branch': 'main', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': '1fb5d10608dccee73edd6ef8a2c7b804a097c324', 'targets': {'smithi086.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKxPlMYyYmBRZDXnysowdOFSkkk1PV8lkGFIQWavm/+mvY2O+h5sZqC/0bFk9+9k39a/tF2jq129pkZF9EhNJGE=', 'smithi196.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBITeA4FH29GBMbjGs2YpZjmD02Lke+TSUSNBt/bSNE5Jj2yN5L+e0jVZLnzu8Z5jXcWb02DMbALytQ3wv0SU21k='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': None}, {'workunit': {'clients': {'client.0': ['rados/test.sh']}, 'env': {'ALLOW_TIMEOUTS': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': 'e691533f9cbb33d85b2187bba20d7102f098636d', 'timestamp': '2024-03-26_05:45:44', 'tube': 'smithi', 'user': 'amathuri', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-03-29T11:19:53.387 DEBUG:teuthology.orchestra.run.smithi086:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:19:53.416 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-03-29T11:19:53.430 DEBUG:teuthology.orchestra.run.smithi086:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:19:53.471 DEBUG:teuthology.orchestra.run.smithi196:> sudo install -d -m0777 -- /var/run/ceph 2024-03-29T11:19:53.498 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T11:19:53.498 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:19:53.550 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:19:53.550 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_1 2024-03-29T11:19:53.604 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:19:53.605 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:19:53.605 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-29T11:19:53.605 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:19:53.605 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:19:53.605 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-03-29 11:19:10.438439370 +0000 2024-03-29T11:19:53.605 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-03-29 11:15:44.647430609 +0000 2024-03-29T11:19:53.605 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-03-29 11:15:44.647430609 +0000 2024-03-29T11:19:53.605 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-03-29 11:15:44.647430609 +0000 2024-03-29T11:19:53.605 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:19:53.670 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-03-29T11:19:53.670 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-03-29T11:19:53.670 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.00020208 s, 2.5 MB/s 2024-03-29T11:19:53.671 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:19:53.727 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_2 2024-03-29T11:19:53.782 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:19:53.782 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:19:53.782 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-29T11:19:53.782 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:19:53.782 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:19:53.782 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-03-29 11:19:10.439439348 +0000 2024-03-29T11:19:53.782 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-03-29 11:15:44.971423237 +0000 2024-03-29T11:19:53.783 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-03-29 11:15:44.971423237 +0000 2024-03-29T11:19:53.783 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-03-29 11:15:44.971423237 +0000 2024-03-29T11:19:53.783 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:19:53.847 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-03-29T11:19:53.847 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-03-29T11:19:53.847 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000200412 s, 2.6 MB/s 2024-03-29T11:19:53.848 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:19:53.904 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_3 2024-03-29T11:19:53.957 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:19:53.958 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:19:53.958 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-29T11:19:53.958 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:19:53.958 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:19:53.958 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-03-29 11:19:10.439439348 +0000 2024-03-29T11:19:53.958 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-03-29 11:15:45.313415457 +0000 2024-03-29T11:19:53.958 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-03-29 11:15:45.313415457 +0000 2024-03-29T11:19:53.958 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-03-29 11:15:45.313415457 +0000 2024-03-29T11:19:53.958 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:19:54.023 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-03-29T11:19:54.024 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-03-29T11:19:54.024 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000196417 s, 2.6 MB/s 2024-03-29T11:19:54.025 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:19:54.080 DEBUG:teuthology.orchestra.run.smithi086:> stat /dev/vg_nvme/lv_4 2024-03-29T11:19:54.135 INFO:teuthology.orchestra.run.smithi086.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:19:54.135 INFO:teuthology.orchestra.run.smithi086.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:19:54.135 INFO:teuthology.orchestra.run.smithi086.stdout:Device: 5h/5d Inode: 724 Links: 1 2024-03-29T11:19:54.135 INFO:teuthology.orchestra.run.smithi086.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:19:54.135 INFO:teuthology.orchestra.run.smithi086.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:19:54.136 INFO:teuthology.orchestra.run.smithi086.stdout:Access: 2024-03-29 11:19:10.439439348 +0000 2024-03-29T11:19:54.136 INFO:teuthology.orchestra.run.smithi086.stdout:Modify: 2024-03-29 11:15:45.644407926 +0000 2024-03-29T11:19:54.136 INFO:teuthology.orchestra.run.smithi086.stdout:Change: 2024-03-29 11:15:45.644407926 +0000 2024-03-29T11:19:54.136 INFO:teuthology.orchestra.run.smithi086.stdout: Birth: 2024-03-29 11:15:45.644407926 +0000 2024-03-29T11:19:54.136 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:19:54.201 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records in 2024-03-29T11:19:54.201 INFO:teuthology.orchestra.run.smithi086.stderr:1+0 records out 2024-03-29T11:19:54.202 INFO:teuthology.orchestra.run.smithi086.stderr:512 bytes copied, 0.000191253 s, 2.7 MB/s 2024-03-29T11:19:54.202 DEBUG:teuthology.orchestra.run.smithi086:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:19:54.258 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:54.258 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T11:19:54.258 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/scratch_devs of=/dev/stdout 2024-03-29T11:19:54.274 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-29T11:19:54.275 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_1 2024-03-29T11:19:54.333 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-29T11:19:54.333 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:19:54.333 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-29T11:19:54.334 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:19:54.334 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:19:54.334 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-03-29 11:19:48.579902095 +0000 2024-03-29T11:19:54.334 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-03-29 11:15:52.536097421 +0000 2024-03-29T11:19:54.334 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-03-29 11:15:52.536097421 +0000 2024-03-29T11:19:54.334 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-03-29 11:15:52.536097421 +0000 2024-03-29T11:19:54.334 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-29T11:19:54.400 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-03-29T11:19:54.401 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-03-29T11:19:54.401 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000250505 s, 2.0 MB/s 2024-03-29T11:19:54.402 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-29T11:19:54.458 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_2 2024-03-29T11:19:54.513 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-29T11:19:54.513 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:19:54.513 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-29T11:19:54.513 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:19:54.513 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:19:54.514 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-03-29 11:19:48.580902074 +0000 2024-03-29T11:19:54.514 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-03-29 11:15:52.880090211 +0000 2024-03-29T11:19:54.514 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-03-29 11:15:52.880090211 +0000 2024-03-29T11:19:54.514 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-03-29 11:15:52.880090211 +0000 2024-03-29T11:19:54.515 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-29T11:19:54.579 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-03-29T11:19:54.580 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-03-29T11:19:54.580 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000205215 s, 2.5 MB/s 2024-03-29T11:19:54.581 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-29T11:19:54.637 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_3 2024-03-29T11:19:54.692 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-29T11:19:54.692 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:19:54.692 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 706 Links: 1 2024-03-29T11:19:54.692 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:19:54.693 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:19:54.693 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-03-29 11:19:48.580902074 +0000 2024-03-29T11:19:54.693 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-03-29 11:15:53.231082855 +0000 2024-03-29T11:19:54.693 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-03-29 11:15:53.231082855 +0000 2024-03-29T11:19:54.693 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-03-29 11:15:53.231082855 +0000 2024-03-29T11:19:54.693 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-29T11:19:54.758 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-03-29T11:19:54.758 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-03-29T11:19:54.758 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000207667 s, 2.5 MB/s 2024-03-29T11:19:54.759 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-29T11:19:54.815 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_4 2024-03-29T11:19:54.870 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-29T11:19:54.870 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-29T11:19:54.870 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-03-29T11:19:54.870 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-29T11:19:54.870 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-03-29T11:19:54.871 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-03-29 11:19:48.580902074 +0000 2024-03-29T11:19:54.871 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-03-29 11:15:53.566075835 +0000 2024-03-29T11:19:54.871 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-03-29 11:15:53.566075835 +0000 2024-03-29T11:19:54.871 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-03-29 11:15:53.566075835 +0000 2024-03-29T11:19:54.871 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-29T11:19:54.936 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-03-29T11:19:54.936 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-03-29T11:19:54.936 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000203517 s, 2.5 MB/s 2024-03-29T11:19:54.937 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-29T11:19:54.992 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:54.993 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi086.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi196.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:19:54.993 INFO:tasks.ceph:Generating config... 2024-03-29T11:19:54.994 INFO:tasks.ceph:[client] debug monc = 20 2024-03-29T11:19:54.994 INFO:tasks.ceph:[client] debug ms = 1 2024-03-29T11:19:54.994 INFO:tasks.ceph:[client] debug objecter = 20 2024-03-29T11:19:54.994 INFO:tasks.ceph:[client] debug rados = 20 2024-03-29T11:19:54.994 INFO:tasks.ceph:[global] lockdep = True 2024-03-29T11:19:54.994 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-03-29T11:19:54.994 INFO:tasks.ceph:[global] mon election default strategy = 1 2024-03-29T11:19:54.994 INFO:tasks.ceph:[global] ms bind msgr2 = False 2024-03-29T11:19:54.994 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-03-29T11:19:54.994 INFO:tasks.ceph:[global] ms type = async 2024-03-29T11:19:54.995 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-03-29T11:19:54.995 INFO:tasks.ceph:[mgr] debug ms = 1 2024-03-29T11:19:54.995 INFO:tasks.ceph:[mon] debug mon = 20 2024-03-29T11:19:54.995 INFO:tasks.ceph:[mon] debug ms = 1 2024-03-29T11:19:54.995 INFO:tasks.ceph:[mon] debug paxos = 20 2024-03-29T11:19:54.995 INFO:tasks.ceph:[mon] mon min osdmap epochs = 50 2024-03-29T11:19:54.995 INFO:tasks.ceph:[mon] mon osdmap full prune interval = 2 2024-03-29T11:19:54.995 INFO:tasks.ceph:[mon] mon osdmap full prune min = 15 2024-03-29T11:19:54.995 INFO:tasks.ceph:[mon] mon osdmap full prune txsize = 2 2024-03-29T11:19:54.995 INFO:tasks.ceph:[mon] mon scrub interval = 300 2024-03-29T11:19:54.995 INFO:tasks.ceph:[mon] mon warn on pool no app = False 2024-03-29T11:19:54.995 INFO:tasks.ceph:[mon] paxos service trim min = 10 2024-03-29T11:19:54.995 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-03-29T11:19:54.995 INFO:tasks.ceph:[osd] bluestore compression algorithm = snappy 2024-03-29T11:19:54.996 INFO:tasks.ceph:[osd] bluestore compression mode = aggressive 2024-03-29T11:19:54.996 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-03-29T11:19:54.996 INFO:tasks.ceph:[osd] bluestore zero block detection = True 2024-03-29T11:19:54.996 INFO:tasks.ceph:[osd] debug bluefs = 20 2024-03-29T11:19:54.996 INFO:tasks.ceph:[osd] debug bluestore = 20 2024-03-29T11:19:54.996 INFO:tasks.ceph:[osd] debug monc = 20 2024-03-29T11:19:54.996 INFO:tasks.ceph:[osd] debug ms = 1 2024-03-29T11:19:54.996 INFO:tasks.ceph:[osd] debug osd = 20 2024-03-29T11:19:54.996 INFO:tasks.ceph:[osd] debug rocksdb = 10 2024-03-29T11:19:54.996 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-03-29T11:19:54.996 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-03-29T11:19:54.996 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-03-29T11:19:54.996 INFO:tasks.ceph:[osd] osd class default list = * 2024-03-29T11:19:54.996 INFO:tasks.ceph:[osd] osd class load list = * 2024-03-29T11:19:54.996 INFO:tasks.ceph:[osd] osd client watch timeout = 120 2024-03-29T11:19:54.997 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2024-03-29T11:19:54.997 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2024-03-29T11:19:54.997 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-03-29T11:19:54.997 INFO:tasks.ceph:[osd] osd mclock override recovery settings = True 2024-03-29T11:19:54.997 INFO:tasks.ceph:[osd] osd mclock profile = high_recovery_ops 2024-03-29T11:19:54.997 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-03-29T11:19:54.997 INFO:tasks.ceph:[osd] osd op queue = debug_random 2024-03-29T11:19:54.997 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2024-03-29T11:19:54.997 INFO:tasks.ceph:[osd] osd shutdown pgref assert = True 2024-03-29T11:19:54.997 INFO:tasks.ceph:Setting up mon.a... 2024-03-29T11:19:54.997 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-03-29T11:19:55.058 INFO:teuthology.orchestra.run.smithi086.stdout:creating /etc/ceph/ceph.keyring 2024-03-29T11:19:55.061 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-03-29T11:19:55.108 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:19:55.135 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.86:6789]'), ('mon.c', '[v1:172.21.15.86:6790]'), ('mon.b', '[v1:172.21.15.196:6789]')] 2024-03-29T11:19:55.136 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.86:6789],[v1:172.21.15.86:6790],[v1:172.21.15.196:6789]', 'lockdep': True, 'mon client directed command retry': 5, 'mon election default strategy': 1, 'ms bind msgr2': False, 'ms inject socket failures': 5000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bluestore block size': 96636764160, 'bluestore compression algorithm': 'snappy', 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'bluestore zero block detection': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug monc': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd class default list': '*', 'osd class load list': '*', 'osd client watch timeout': 120, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd mclock override recovery settings': True, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random', 'osd shutdown pgref assert': True}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'mon scrub interval': 300, 'mon warn on pool no app': False, 'paxos service trim min': 10}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug monc': 20, 'debug ms': 1, 'debug objecter': 20, 'debug rados': 20}, 'mon.a': {}, 'mon.c': {}, 'mon.b': {}} 2024-03-29T11:19:55.137 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T11:19:55.137 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:19:55.191 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.86:6789]' --addv c '[v1:172.21.15.86:6790]' --addv b '[v1:172.21.15.196:6789]' --print /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:55.277 INFO:teuthology.orchestra.run.smithi086.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:55.277 INFO:teuthology.orchestra.run.smithi086.stdout:monmaptool: generated fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:19:55.277 INFO:teuthology.orchestra.run.smithi086.stdout:setting min_mon_release = quincy 2024-03-29T11:19:55.277 INFO:teuthology.orchestra.run.smithi086.stdout:epoch 0 2024-03-29T11:19:55.277 INFO:teuthology.orchestra.run.smithi086.stdout:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:19:55.277 INFO:teuthology.orchestra.run.smithi086.stdout:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:19:55.277 INFO:teuthology.orchestra.run.smithi086.stdout:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:19:55.277 INFO:teuthology.orchestra.run.smithi086.stdout:min_mon_release 17 (quincy) 2024-03-29T11:19:55.278 INFO:teuthology.orchestra.run.smithi086.stdout:election_strategy: 1 2024-03-29T11:19:55.278 INFO:teuthology.orchestra.run.smithi086.stdout:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:19:55.278 INFO:teuthology.orchestra.run.smithi086.stdout:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:19:55.278 INFO:teuthology.orchestra.run.smithi086.stdout:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:19:55.278 INFO:teuthology.orchestra.run.smithi086.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-03-29T11:19:55.279 DEBUG:teuthology.orchestra.run.smithi086:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-03-29T11:19:55.294 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 12b413e9-4891-4542-8b6d-70acb06d2a3d... 2024-03-29T11:19:55.295 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:19:55.336 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-03-29T11:19:55.383 INFO:teuthology.orchestra.run.smithi086.stdout:[global] 2024-03-29T11:19:55.383 INFO:teuthology.orchestra.run.smithi086.stdout: chdir = "" 2024-03-29T11:19:55.383 INFO:teuthology.orchestra.run.smithi086.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:19:55.383 INFO:teuthology.orchestra.run.smithi086.stdout: auth supported = cephx 2024-03-29T11:19:55.384 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:55.384 INFO:teuthology.orchestra.run.smithi086.stdout: filestore xattr use omap = true 2024-03-29T11:19:55.384 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:55.384 INFO:teuthology.orchestra.run.smithi086.stdout: mon clock drift allowed = 1.000 2024-03-29T11:19:55.384 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:55.384 INFO:teuthology.orchestra.run.smithi086.stdout: osd crush chooseleaf type = 0 2024-03-29T11:19:55.384 INFO:teuthology.orchestra.run.smithi086.stdout: auth debug = true 2024-03-29T11:19:55.384 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:55.384 INFO:teuthology.orchestra.run.smithi086.stdout: ms die on old message = true 2024-03-29T11:19:55.384 INFO:teuthology.orchestra.run.smithi086.stdout: ms die on bug = true 2024-03-29T11:19:55.384 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:55.385 INFO:teuthology.orchestra.run.smithi086.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:19:55.385 INFO:teuthology.orchestra.run.smithi086.stdout: mon pg warn max object skew = 0 2024-03-29T11:19:55.385 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:55.385 INFO:teuthology.orchestra.run.smithi086.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:19:55.385 INFO:teuthology.orchestra.run.smithi086.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:19:55.385 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:55.385 INFO:teuthology.orchestra.run.smithi086.stdout: osd pool default size = 2 2024-03-29T11:19:55.385 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:55.385 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd allow primary affinity = true 2024-03-29T11:19:55.385 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd allow pg remap = true 2024-03-29T11:19:55.385 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:19:55.385 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:19:55.385 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on no sortbitwise = false 2024-03-29T11:19:55.385 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:19:55.386 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on too few osds = false 2024-03-29T11:19:55.386 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:19:55.386 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:19:55.386 INFO:teuthology.orchestra.run.smithi086.stdout: mon_allow_pool_size_one = true 2024-03-29T11:19:55.386 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:55.386 INFO:teuthology.orchestra.run.smithi086.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:19:55.386 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:55.386 INFO:teuthology.orchestra.run.smithi086.stdout: osd default data pool replay window = 5 2024-03-29T11:19:55.386 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:55.386 INFO:teuthology.orchestra.run.smithi086.stdout: mon allow pool delete = true 2024-03-29T11:19:55.386 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:55.386 INFO:teuthology.orchestra.run.smithi086.stdout: mon cluster log file level = debug 2024-03-29T11:19:55.386 INFO:teuthology.orchestra.run.smithi086.stdout: debug asserts on shutdown = true 2024-03-29T11:19:55.387 INFO:teuthology.orchestra.run.smithi086.stdout: mon health detail to clog = false 2024-03-29T11:19:55.387 INFO:teuthology.orchestra.run.smithi086.stdout: mon host = "[v1:172.21.15.86:6789],[v1:172.21.15.86:6790],[v1:172.21.15.196:6789]" 2024-03-29T11:19:55.387 INFO:teuthology.orchestra.run.smithi086.stdout: lockdep = True 2024-03-29T11:19:55.387 INFO:teuthology.orchestra.run.smithi086.stdout: mon client directed command retry = 5 2024-03-29T11:19:55.387 INFO:teuthology.orchestra.run.smithi086.stdout: mon election default strategy = 1 2024-03-29T11:19:55.387 INFO:teuthology.orchestra.run.smithi086.stdout: ms bind msgr2 = False 2024-03-29T11:19:55.387 INFO:teuthology.orchestra.run.smithi086.stdout: ms inject socket failures = 5000 2024-03-29T11:19:55.387 INFO:teuthology.orchestra.run.smithi086.stdout: ms type = async 2024-03-29T11:19:55.387 INFO:teuthology.orchestra.run.smithi086.stdout: fsid = 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:19:55.387 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:55.387 INFO:teuthology.orchestra.run.smithi086.stdout:[osd] 2024-03-29T11:19:55.387 INFO:teuthology.orchestra.run.smithi086.stdout: osd journal size = 100 2024-03-29T11:19:55.387 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:55.387 INFO:teuthology.orchestra.run.smithi086.stdout: osd scrub load threshold = 5.0 2024-03-29T11:19:55.387 INFO:teuthology.orchestra.run.smithi086.stdout: osd scrub max interval = 600 2024-03-29T11:19:55.388 INFO:teuthology.orchestra.run.smithi086.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:19:55.388 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:55.388 INFO:teuthology.orchestra.run.smithi086.stdout: osd recover clone overlap = true 2024-03-29T11:19:55.388 INFO:teuthology.orchestra.run.smithi086.stdout: osd recovery max chunk = 1048576 2024-03-29T11:19:55.388 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:55.388 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug shutdown = true 2024-03-29T11:19:55.388 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug op order = true 2024-03-29T11:19:55.388 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug verify stray on activate = true 2024-03-29T11:19:55.388 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug trim objects = true 2024-03-29T11:19:55.388 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:55.388 INFO:teuthology.orchestra.run.smithi086.stdout: osd open classes on start = true 2024-03-29T11:19:55.388 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug pg log writeout = true 2024-03-29T11:19:55.388 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:55.388 INFO:teuthology.orchestra.run.smithi086.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:19:55.388 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:55.389 INFO:teuthology.orchestra.run.smithi086.stdout: osd map max advance = 10 2024-03-29T11:19:55.389 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:55.389 INFO:teuthology.orchestra.run.smithi086.stdout: journal zero on create = true 2024-03-29T11:19:55.389 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:55.389 INFO:teuthology.orchestra.run.smithi086.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:19:55.389 INFO:teuthology.orchestra.run.smithi086.stdout: filestore apply finisher threads = 3 2024-03-29T11:19:55.389 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:55.389 INFO:teuthology.orchestra.run.smithi086.stdout: bdev debug aio = true 2024-03-29T11:19:55.389 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug misdirected ops = true 2024-03-29T11:19:55.389 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore block size = 96636764160 2024-03-29T11:19:55.389 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore compression algorithm = snappy 2024-03-29T11:19:55.389 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore compression mode = aggressive 2024-03-29T11:19:55.389 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore fsck on mount = True 2024-03-29T11:19:55.390 INFO:teuthology.orchestra.run.smithi086.stdout: bluestore zero block detection = True 2024-03-29T11:19:55.390 INFO:teuthology.orchestra.run.smithi086.stdout: debug bluefs = 20 2024-03-29T11:19:55.390 INFO:teuthology.orchestra.run.smithi086.stdout: debug bluestore = 20 2024-03-29T11:19:55.390 INFO:teuthology.orchestra.run.smithi086.stdout: debug monc = 20 2024-03-29T11:19:55.390 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2024-03-29T11:19:55.390 INFO:teuthology.orchestra.run.smithi086.stdout: debug osd = 20 2024-03-29T11:19:55.390 INFO:teuthology.orchestra.run.smithi086.stdout: debug rocksdb = 10 2024-03-29T11:19:55.390 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:19:55.390 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd full ratio = 0.9 2024-03-29T11:19:55.390 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:19:55.390 INFO:teuthology.orchestra.run.smithi086.stdout: osd class default list = * 2024-03-29T11:19:55.390 INFO:teuthology.orchestra.run.smithi086.stdout: osd class load list = * 2024-03-29T11:19:55.390 INFO:teuthology.orchestra.run.smithi086.stdout: osd client watch timeout = 120 2024-03-29T11:19:55.390 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug verify cached snaps = True 2024-03-29T11:19:55.390 INFO:teuthology.orchestra.run.smithi086.stdout: osd debug verify missing on start = True 2024-03-29T11:19:55.391 INFO:teuthology.orchestra.run.smithi086.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:19:55.391 INFO:teuthology.orchestra.run.smithi086.stdout: osd mclock override recovery settings = True 2024-03-29T11:19:55.391 INFO:teuthology.orchestra.run.smithi086.stdout: osd objectstore = bluestore 2024-03-29T11:19:55.391 INFO:teuthology.orchestra.run.smithi086.stdout: osd op queue = debug_random 2024-03-29T11:19:55.391 INFO:teuthology.orchestra.run.smithi086.stdout: osd op queue cut off = debug_random 2024-03-29T11:19:55.391 INFO:teuthology.orchestra.run.smithi086.stdout: osd shutdown pgref assert = True 2024-03-29T11:19:55.391 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:55.391 INFO:teuthology.orchestra.run.smithi086.stdout:[mgr] 2024-03-29T11:19:55.391 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2024-03-29T11:19:55.391 INFO:teuthology.orchestra.run.smithi086.stdout: debug mgr = 20 2024-03-29T11:19:55.391 INFO:teuthology.orchestra.run.smithi086.stdout: debug mon = 20 2024-03-29T11:19:55.391 INFO:teuthology.orchestra.run.smithi086.stdout: debug auth = 20 2024-03-29T11:19:55.391 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:19:55.392 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:19:55.392 INFO:teuthology.orchestra.run.smithi086.stdout: mgr/telemetry/nag = false 2024-03-29T11:19:55.392 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:55.392 INFO:teuthology.orchestra.run.smithi086.stdout:[mon] 2024-03-29T11:19:55.392 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2024-03-29T11:19:55.392 INFO:teuthology.orchestra.run.smithi086.stdout: debug mon = 20 2024-03-29T11:19:55.392 INFO:teuthology.orchestra.run.smithi086.stdout: debug paxos = 20 2024-03-29T11:19:55.392 INFO:teuthology.orchestra.run.smithi086.stdout: debug auth = 20 2024-03-29T11:19:55.392 INFO:teuthology.orchestra.run.smithi086.stdout: mon data avail warn = 5 2024-03-29T11:19:55.392 INFO:teuthology.orchestra.run.smithi086.stdout: mon mgr mkfs grace = 240 2024-03-29T11:19:55.392 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:19:55.392 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd reporter subtree level = osd 2024-03-29T11:19:55.392 INFO:teuthology.orchestra.run.smithi086.stdout: mon osd prime pg temp = true 2024-03-29T11:19:55.392 INFO:teuthology.orchestra.run.smithi086.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:19:55.392 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:55.393 INFO:teuthology.orchestra.run.smithi086.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:19:55.393 INFO:teuthology.orchestra.run.smithi086.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:19:55.393 INFO:teuthology.orchestra.run.smithi086.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:19:55.393 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:55.393 INFO:teuthology.orchestra.run.smithi086.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:19:55.393 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:19:55.393 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:19:55.393 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:55.393 INFO:teuthology.orchestra.run.smithi086.stdout: # 1m isn't quite enough 2024-03-29T11:19:55.393 INFO:teuthology.orchestra.run.smithi086.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:19:55.393 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:55.393 INFO:teuthology.orchestra.run.smithi086.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:19:55.393 INFO:teuthology.orchestra.run.smithi086.stdout: mon min osdmap epochs = 50 2024-03-29T11:19:55.393 INFO:teuthology.orchestra.run.smithi086.stdout: mon osdmap full prune interval = 2 2024-03-29T11:19:55.393 INFO:teuthology.orchestra.run.smithi086.stdout: mon osdmap full prune min = 15 2024-03-29T11:19:55.394 INFO:teuthology.orchestra.run.smithi086.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:19:55.394 INFO:teuthology.orchestra.run.smithi086.stdout: mon scrub interval = 300 2024-03-29T11:19:55.394 INFO:teuthology.orchestra.run.smithi086.stdout: mon warn on pool no app = False 2024-03-29T11:19:55.394 INFO:teuthology.orchestra.run.smithi086.stdout: paxos service trim min = 10 2024-03-29T11:19:55.394 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:19:55.394 INFO:teuthology.orchestra.run.smithi086.stdout:[client] 2024-03-29T11:19:55.394 INFO:teuthology.orchestra.run.smithi086.stdout: rgw cache enabled = true 2024-03-29T11:19:55.394 INFO:teuthology.orchestra.run.smithi086.stdout: rgw enable ops log = true 2024-03-29T11:19:55.394 INFO:teuthology.orchestra.run.smithi086.stdout: rgw enable usage log = true 2024-03-29T11:19:55.394 INFO:teuthology.orchestra.run.smithi086.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:19:55.394 INFO:teuthology.orchestra.run.smithi086.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:19:55.394 INFO:teuthology.orchestra.run.smithi086.stdout: debug monc = 20 2024-03-29T11:19:55.394 INFO:teuthology.orchestra.run.smithi086.stdout: debug ms = 1 2024-03-29T11:19:55.394 INFO:teuthology.orchestra.run.smithi086.stdout: debug objecter = 20 2024-03-29T11:19:55.394 INFO:teuthology.orchestra.run.smithi086.stdout: debug rados = 20 2024-03-29T11:19:55.395 INFO:teuthology.orchestra.run.smithi086.stdout:[mon.a] 2024-03-29T11:19:55.395 INFO:teuthology.orchestra.run.smithi086.stdout:[mon.c] 2024-03-29T11:19:55.395 INFO:teuthology.orchestra.run.smithi086.stdout:[mon.b] 2024-03-29T11:19:55.395 INFO:teuthology.orchestra.run.smithi196.stdout:[global] 2024-03-29T11:19:55.395 INFO:teuthology.orchestra.run.smithi196.stdout: chdir = "" 2024-03-29T11:19:55.395 INFO:teuthology.orchestra.run.smithi196.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-03-29T11:19:55.395 INFO:teuthology.orchestra.run.smithi196.stdout: auth supported = cephx 2024-03-29T11:19:55.395 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:55.395 INFO:teuthology.orchestra.run.smithi196.stdout: filestore xattr use omap = true 2024-03-29T11:19:55.395 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:55.396 INFO:teuthology.orchestra.run.smithi196.stdout: mon clock drift allowed = 1.000 2024-03-29T11:19:55.396 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:55.396 INFO:teuthology.orchestra.run.smithi196.stdout: osd crush chooseleaf type = 0 2024-03-29T11:19:55.396 INFO:teuthology.orchestra.run.smithi196.stdout: auth debug = true 2024-03-29T11:19:55.396 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:55.396 INFO:teuthology.orchestra.run.smithi196.stdout: ms die on old message = true 2024-03-29T11:19:55.396 INFO:teuthology.orchestra.run.smithi196.stdout: ms die on bug = true 2024-03-29T11:19:55.396 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:55.396 INFO:teuthology.orchestra.run.smithi196.stdout: mon max pg per osd = 10000 # >= luminous 2024-03-29T11:19:55.396 INFO:teuthology.orchestra.run.smithi196.stdout: mon pg warn max object skew = 0 2024-03-29T11:19:55.396 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:55.396 INFO:teuthology.orchestra.run.smithi196.stdout: # disable pg_autoscaler by default for new pools 2024-03-29T11:19:55.396 INFO:teuthology.orchestra.run.smithi196.stdout: osd_pool_default_pg_autoscale_mode = off 2024-03-29T11:19:55.397 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:55.397 INFO:teuthology.orchestra.run.smithi196.stdout: osd pool default size = 2 2024-03-29T11:19:55.397 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:55.397 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd allow primary affinity = true 2024-03-29T11:19:55.397 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd allow pg remap = true 2024-03-29T11:19:55.397 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on legacy crush tunables = false 2024-03-29T11:19:55.397 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on crush straw calc version zero = false 2024-03-29T11:19:55.397 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on no sortbitwise = false 2024-03-29T11:19:55.397 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on osd down out interval zero = false 2024-03-29T11:19:55.397 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on too few osds = false 2024-03-29T11:19:55.397 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-03-29T11:19:55.397 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_pool_no_redundancy = false 2024-03-29T11:19:55.398 INFO:teuthology.orchestra.run.smithi196.stdout: mon_allow_pool_size_one = true 2024-03-29T11:19:55.398 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:55.398 INFO:teuthology.orchestra.run.smithi196.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-03-29T11:19:55.398 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:55.398 INFO:teuthology.orchestra.run.smithi196.stdout: osd default data pool replay window = 5 2024-03-29T11:19:55.398 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:55.398 INFO:teuthology.orchestra.run.smithi196.stdout: mon allow pool delete = true 2024-03-29T11:19:55.398 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:55.398 INFO:teuthology.orchestra.run.smithi196.stdout: mon cluster log file level = debug 2024-03-29T11:19:55.398 INFO:teuthology.orchestra.run.smithi196.stdout: debug asserts on shutdown = true 2024-03-29T11:19:55.398 INFO:teuthology.orchestra.run.smithi196.stdout: mon health detail to clog = false 2024-03-29T11:19:55.398 INFO:teuthology.orchestra.run.smithi196.stdout: mon host = "[v1:172.21.15.86:6789],[v1:172.21.15.86:6790],[v1:172.21.15.196:6789]" 2024-03-29T11:19:55.398 INFO:teuthology.orchestra.run.smithi196.stdout: lockdep = True 2024-03-29T11:19:55.399 INFO:teuthology.orchestra.run.smithi196.stdout: mon client directed command retry = 5 2024-03-29T11:19:55.399 INFO:teuthology.orchestra.run.smithi196.stdout: mon election default strategy = 1 2024-03-29T11:19:55.399 INFO:teuthology.orchestra.run.smithi196.stdout: ms bind msgr2 = False 2024-03-29T11:19:55.399 INFO:teuthology.orchestra.run.smithi196.stdout: ms inject socket failures = 5000 2024-03-29T11:19:55.399 INFO:teuthology.orchestra.run.smithi196.stdout: ms type = async 2024-03-29T11:19:55.399 INFO:teuthology.orchestra.run.smithi196.stdout: fsid = 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:19:55.399 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:55.399 INFO:teuthology.orchestra.run.smithi196.stdout:[osd] 2024-03-29T11:19:55.399 INFO:teuthology.orchestra.run.smithi196.stdout: osd journal size = 100 2024-03-29T11:19:55.399 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:55.399 INFO:teuthology.orchestra.run.smithi196.stdout: osd scrub load threshold = 5.0 2024-03-29T11:19:55.399 INFO:teuthology.orchestra.run.smithi196.stdout: osd scrub max interval = 600 2024-03-29T11:19:55.399 INFO:teuthology.orchestra.run.smithi196.stdout: osd mclock profile = high_recovery_ops 2024-03-29T11:19:55.399 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:55.399 INFO:teuthology.orchestra.run.smithi196.stdout: osd recover clone overlap = true 2024-03-29T11:19:55.400 INFO:teuthology.orchestra.run.smithi196.stdout: osd recovery max chunk = 1048576 2024-03-29T11:19:55.400 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:55.400 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug shutdown = true 2024-03-29T11:19:55.400 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug op order = true 2024-03-29T11:19:55.400 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug verify stray on activate = true 2024-03-29T11:19:55.400 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug trim objects = true 2024-03-29T11:19:55.400 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:55.400 INFO:teuthology.orchestra.run.smithi196.stdout: osd open classes on start = true 2024-03-29T11:19:55.400 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug pg log writeout = true 2024-03-29T11:19:55.400 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:55.400 INFO:teuthology.orchestra.run.smithi196.stdout: osd deep scrub update digest min age = 30 2024-03-29T11:19:55.400 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:55.401 INFO:teuthology.orchestra.run.smithi196.stdout: osd map max advance = 10 2024-03-29T11:19:55.401 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:55.401 INFO:teuthology.orchestra.run.smithi196.stdout: journal zero on create = true 2024-03-29T11:19:55.401 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:55.401 INFO:teuthology.orchestra.run.smithi196.stdout: filestore ondisk finisher threads = 3 2024-03-29T11:19:55.401 INFO:teuthology.orchestra.run.smithi196.stdout: filestore apply finisher threads = 3 2024-03-29T11:19:55.401 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:55.401 INFO:teuthology.orchestra.run.smithi196.stdout: bdev debug aio = true 2024-03-29T11:19:55.401 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug misdirected ops = true 2024-03-29T11:19:55.401 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore block size = 96636764160 2024-03-29T11:19:55.401 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore compression algorithm = snappy 2024-03-29T11:19:55.401 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore compression mode = aggressive 2024-03-29T11:19:55.401 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore fsck on mount = True 2024-03-29T11:19:55.401 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore zero block detection = True 2024-03-29T11:19:55.402 INFO:teuthology.orchestra.run.smithi196.stdout: debug bluefs = 20 2024-03-29T11:19:55.402 INFO:teuthology.orchestra.run.smithi196.stdout: debug bluestore = 20 2024-03-29T11:19:55.402 INFO:teuthology.orchestra.run.smithi196.stdout: debug monc = 20 2024-03-29T11:19:55.402 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2024-03-29T11:19:55.402 INFO:teuthology.orchestra.run.smithi196.stdout: debug osd = 20 2024-03-29T11:19:55.402 INFO:teuthology.orchestra.run.smithi196.stdout: debug rocksdb = 10 2024-03-29T11:19:55.402 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd backfillfull_ratio = 0.85 2024-03-29T11:19:55.402 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd full ratio = 0.9 2024-03-29T11:19:55.402 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd nearfull ratio = 0.8 2024-03-29T11:19:55.402 INFO:teuthology.orchestra.run.smithi196.stdout: osd class default list = * 2024-03-29T11:19:55.402 INFO:teuthology.orchestra.run.smithi196.stdout: osd class load list = * 2024-03-29T11:19:55.402 INFO:teuthology.orchestra.run.smithi196.stdout: osd client watch timeout = 120 2024-03-29T11:19:55.402 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug verify cached snaps = True 2024-03-29T11:19:55.402 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug verify missing on start = True 2024-03-29T11:19:55.403 INFO:teuthology.orchestra.run.smithi196.stdout: osd failsafe full ratio = 0.95 2024-03-29T11:19:55.403 INFO:teuthology.orchestra.run.smithi196.stdout: osd mclock override recovery settings = True 2024-03-29T11:19:55.403 INFO:teuthology.orchestra.run.smithi196.stdout: osd objectstore = bluestore 2024-03-29T11:19:55.403 INFO:teuthology.orchestra.run.smithi196.stdout: osd op queue = debug_random 2024-03-29T11:19:55.403 INFO:teuthology.orchestra.run.smithi196.stdout: osd op queue cut off = debug_random 2024-03-29T11:19:55.403 INFO:teuthology.orchestra.run.smithi196.stdout: osd shutdown pgref assert = True 2024-03-29T11:19:55.403 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:55.403 INFO:teuthology.orchestra.run.smithi196.stdout:[mgr] 2024-03-29T11:19:55.403 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2024-03-29T11:19:55.403 INFO:teuthology.orchestra.run.smithi196.stdout: debug mgr = 20 2024-03-29T11:19:55.403 INFO:teuthology.orchestra.run.smithi196.stdout: debug mon = 20 2024-03-29T11:19:55.403 INFO:teuthology.orchestra.run.smithi196.stdout: debug auth = 20 2024-03-29T11:19:55.403 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:19:55.403 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:19:55.403 INFO:teuthology.orchestra.run.smithi196.stdout: mgr/telemetry/nag = false 2024-03-29T11:19:55.404 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:55.404 INFO:teuthology.orchestra.run.smithi196.stdout:[mon] 2024-03-29T11:19:55.404 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2024-03-29T11:19:55.404 INFO:teuthology.orchestra.run.smithi196.stdout: debug mon = 20 2024-03-29T11:19:55.404 INFO:teuthology.orchestra.run.smithi196.stdout: debug paxos = 20 2024-03-29T11:19:55.404 INFO:teuthology.orchestra.run.smithi196.stdout: debug auth = 20 2024-03-29T11:19:55.404 INFO:teuthology.orchestra.run.smithi196.stdout: mon data avail warn = 5 2024-03-29T11:19:55.404 INFO:teuthology.orchestra.run.smithi196.stdout: mon mgr mkfs grace = 240 2024-03-29T11:19:55.404 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min pgs per osd = 4 2024-03-29T11:19:55.404 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd reporter subtree level = osd 2024-03-29T11:19:55.404 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd prime pg temp = true 2024-03-29T11:19:55.404 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min bytes per osd = 10 2024-03-29T11:19:55.404 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:55.404 INFO:teuthology.orchestra.run.smithi196.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-03-29T11:19:55.404 INFO:teuthology.orchestra.run.smithi196.stdout: auth mon ticket ttl = 660 # 11m 2024-03-29T11:19:55.404 INFO:teuthology.orchestra.run.smithi196.stdout: auth service ticket ttl = 240 # 4m 2024-03-29T11:19:55.405 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:55.405 INFO:teuthology.orchestra.run.smithi196.stdout: # don't complain about insecure global_id in the test suite 2024-03-29T11:19:55.405 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-03-29T11:19:55.405 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-03-29T11:19:55.405 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:55.405 INFO:teuthology.orchestra.run.smithi196.stdout: # 1m isn't quite enough 2024-03-29T11:19:55.405 INFO:teuthology.orchestra.run.smithi196.stdout: mon_down_mkfs_grace = 2m 2024-03-29T11:19:55.405 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:55.405 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_filestore_osds = false 2024-03-29T11:19:55.405 INFO:teuthology.orchestra.run.smithi196.stdout: mon min osdmap epochs = 50 2024-03-29T11:19:55.405 INFO:teuthology.orchestra.run.smithi196.stdout: mon osdmap full prune interval = 2 2024-03-29T11:19:55.405 INFO:teuthology.orchestra.run.smithi196.stdout: mon osdmap full prune min = 15 2024-03-29T11:19:55.405 INFO:teuthology.orchestra.run.smithi196.stdout: mon osdmap full prune txsize = 2 2024-03-29T11:19:55.405 INFO:teuthology.orchestra.run.smithi196.stdout: mon scrub interval = 300 2024-03-29T11:19:55.405 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on pool no app = False 2024-03-29T11:19:55.406 INFO:teuthology.orchestra.run.smithi196.stdout: paxos service trim min = 10 2024-03-29T11:19:55.406 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-03-29T11:19:55.406 INFO:teuthology.orchestra.run.smithi196.stdout:[client] 2024-03-29T11:19:55.406 INFO:teuthology.orchestra.run.smithi196.stdout: rgw cache enabled = true 2024-03-29T11:19:55.406 INFO:teuthology.orchestra.run.smithi196.stdout: rgw enable ops log = true 2024-03-29T11:19:55.406 INFO:teuthology.orchestra.run.smithi196.stdout: rgw enable usage log = true 2024-03-29T11:19:55.406 INFO:teuthology.orchestra.run.smithi196.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-03-29T11:19:55.406 INFO:teuthology.orchestra.run.smithi196.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-03-29T11:19:55.406 INFO:teuthology.orchestra.run.smithi196.stdout: debug monc = 20 2024-03-29T11:19:55.406 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2024-03-29T11:19:55.406 INFO:teuthology.orchestra.run.smithi196.stdout: debug objecter = 20 2024-03-29T11:19:55.406 INFO:teuthology.orchestra.run.smithi196.stdout: debug rados = 20 2024-03-29T11:19:55.406 INFO:teuthology.orchestra.run.smithi196.stdout:[mon.a] 2024-03-29T11:19:55.406 INFO:teuthology.orchestra.run.smithi196.stdout:[mon.c] 2024-03-29T11:19:55.406 INFO:teuthology.orchestra.run.smithi196.stdout:[mon.b] 2024-03-29T11:19:55.408 INFO:tasks.ceph:Creating admin key on mon.a... 2024-03-29T11:19:55.408 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-03-29T11:19:55.503 INFO:tasks.ceph:Copying monmap to all nodes... 2024-03-29T11:19:55.503 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T11:19:55.504 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-03-29T11:19:55.518 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T11:19:55.519 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-03-29T11:19:55.576 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi086.front.sepia.ceph.com 2024-03-29T11:19:55.576 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T11:19:55.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:19:55.576 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:19:55.656 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T11:19:55.656 DEBUG:teuthology.orchestra.run.smithi086:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:55.711 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi196.front.sepia.ceph.com 2024-03-29T11:19:55.711 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T11:19:55.711 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/ceph.keyring 2024-03-29T11:19:55.711 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-03-29T11:19:55.752 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T11:19:55.752 DEBUG:teuthology.orchestra.run.smithi196:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:55.806 INFO:tasks.ceph:Setting up mon nodes... 2024-03-29T11:19:55.806 INFO:tasks.ceph:Setting up mgr nodes... 2024-03-29T11:19:55.807 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:19:55.861 INFO:teuthology.orchestra.run.smithi086.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-03-29T11:19:55.864 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:19:55.934 INFO:teuthology.orchestra.run.smithi196.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-03-29T11:19:55.936 INFO:tasks.ceph:Setting up mds nodes... 2024-03-29T11:19:55.937 INFO:tasks.ceph_client:Setting up client nodes... 2024-03-29T11:19:55.937 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-29T11:19:55.978 INFO:teuthology.orchestra.run.smithi086.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-03-29T11:19:55.994 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-29T11:19:56.036 INFO:teuthology.orchestra.run.smithi196.stdout:creating /etc/ceph/ceph.client.1.keyring 2024-03-29T11:19:56.052 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-03-29T11:19:56.052 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi086.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi196.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-03-29T11:19:56.052 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-03-29T11:19:56.079 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:56.079 INFO:tasks.ceph:role: osd.0 2024-03-29T11:19:56.079 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi086.front.sepia.ceph.com 2024-03-29T11:19:56.080 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:19:56.153 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:56.154 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:56.154 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:56.154 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:56.154 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:56.154 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:56.154 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:56.154 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:56.154 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:56.154 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:56.653 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2024-03-29T11:19:56.655 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2024-03-29T11:19:56.655 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-03-29T11:19:56.784 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-03-29T11:19:56.818 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-03-29T11:19:56.886 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:56.886 INFO:tasks.ceph:role: osd.1 2024-03-29T11:19:56.886 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi086.front.sepia.ceph.com 2024-03-29T11:19:56.887 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:19:56.961 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:56.961 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:56.961 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:56.961 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:56.961 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:56.961 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:56.962 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:56.962 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:56.962 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:56.962 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:57.422 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2024-03-29T11:19:57.424 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2024-03-29T11:19:57.424 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-03-29T11:19:57.523 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-03-29T11:19:57.554 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-03-29T11:19:57.622 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:57.622 INFO:tasks.ceph:role: osd.2 2024-03-29T11:19:57.622 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi086.front.sepia.ceph.com 2024-03-29T11:19:57.622 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:19:57.697 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:57.697 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:57.697 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:57.697 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:57.697 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:57.697 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:57.698 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:57.698 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:57.698 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:57.698 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:58.222 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2024-03-29T11:19:58.224 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2024-03-29T11:19:58.224 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-03-29T11:19:58.356 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-03-29T11:19:58.389 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-03-29T11:19:58.457 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-03-29T11:19:58.457 INFO:tasks.ceph:role: osd.3 2024-03-29T11:19:58.457 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi086.front.sepia.ceph.com 2024-03-29T11:19:58.457 DEBUG:teuthology.orchestra.run.smithi086:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:19:58.531 INFO:teuthology.orchestra.run.smithi086.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:19:58.531 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:19:58.531 INFO:teuthology.orchestra.run.smithi086.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:19:58.531 INFO:teuthology.orchestra.run.smithi086.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:19:58.532 INFO:teuthology.orchestra.run.smithi086.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:19:58.532 INFO:teuthology.orchestra.run.smithi086.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:19:58.532 INFO:teuthology.orchestra.run.smithi086.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:19:58.532 INFO:teuthology.orchestra.run.smithi086.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:19:58.532 INFO:teuthology.orchestra.run.smithi086.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:19:58.532 INFO:teuthology.orchestra.run.smithi086.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:19:59.041 INFO:teuthology.orchestra.run.smithi086.stdout:Discarding blocks...Done. 2024-03-29T11:19:59.043 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi086.front.sepia.ceph.com -o noatime 2024-03-29T11:19:59.043 DEBUG:teuthology.orchestra.run.smithi086:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-03-29T11:19:59.167 DEBUG:teuthology.orchestra.run.smithi086:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-03-29T11:19:59.201 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:19:59.302 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:19:59.324+0000 7f587c613640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-03-29T11:19:59.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:19:59.325+0000 7f587c613640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-03-29T11:19:59.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:19:59.325+0000 7f587c613640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:19:59.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:19:59.325+0000 7f587c613640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:19:59.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:19:59.325+0000 7f587c613640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-03-29T11:19:59.303 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:19:59.325+0000 7f587c613640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-03-29T11:20:02.163 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-03-29T11:20:02.195 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:20:02.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:02.310+0000 7f8f55414640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-03-29T11:20:02.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:02.310+0000 7f8f55414640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-03-29T11:20:02.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:02.310+0000 7f8f55414640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:20:02.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:02.310+0000 7f8f55414640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:20:02.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:02.310+0000 7f8f55414640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-03-29T11:20:02.287 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:02.311+0000 7f8f55414640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-03-29T11:20:05.131 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-03-29T11:20:05.162 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:20:05.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:05.276+0000 7f457201e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-03-29T11:20:05.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:05.276+0000 7f457201e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-03-29T11:20:05.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:05.276+0000 7f457201e640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:20:05.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:05.276+0000 7f457201e640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:20:05.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:05.276+0000 7f457201e640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-03-29T11:20:05.253 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:05.277+0000 7f457201e640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-03-29T11:20:08.110 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-03-29T11:20:08.140 DEBUG:teuthology.orchestra.run.smithi086:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:20:08.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:08.253+0000 7f54fba0d640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-03-29T11:20:08.230 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:08.253+0000 7f54fba0d640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-03-29T11:20:08.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:08.253+0000 7f54fba0d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:20:08.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:08.253+0000 7f54fba0d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:20:08.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:08.253+0000 7f54fba0d640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-03-29T11:20:08.231 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:08.254+0000 7f54fba0d640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-03-29T11:20:11.086 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-03-29T11:20:11.117 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-03-29T11:20:11.150 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:20:11.150 INFO:tasks.ceph:role: osd.4 2024-03-29T11:20:11.150 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi196.front.sepia.ceph.com 2024-03-29T11:20:11.151 DEBUG:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-03-29T11:20:11.228 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:20:11.229 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:20:11.229 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:20:11.229 INFO:teuthology.orchestra.run.smithi196.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:20:11.229 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:20:11.229 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:20:11.229 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:20:11.229 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:20:11.229 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:20:11.229 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:20:11.748 INFO:teuthology.orchestra.run.smithi196.stdout:Discarding blocks...Done. 2024-03-29T11:20:11.750 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2024-03-29T11:20:11.750 DEBUG:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-03-29T11:20:11.876 DEBUG:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-03-29T11:20:11.913 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-03-29T11:20:11.984 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:20:11.984 INFO:tasks.ceph:role: osd.5 2024-03-29T11:20:11.984 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi196.front.sepia.ceph.com 2024-03-29T11:20:11.984 DEBUG:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-03-29T11:20:12.061 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:20:12.061 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:20:12.061 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:20:12.061 INFO:teuthology.orchestra.run.smithi196.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:20:12.061 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:20:12.061 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:20:12.062 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:20:12.062 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:20:12.062 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:20:12.062 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:20:12.511 INFO:teuthology.orchestra.run.smithi196.stdout:Discarding blocks...Done. 2024-03-29T11:20:12.513 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2024-03-29T11:20:12.513 DEBUG:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-03-29T11:20:12.605 DEBUG:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-03-29T11:20:12.636 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-03-29T11:20:12.704 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:20:12.704 INFO:tasks.ceph:role: osd.6 2024-03-29T11:20:12.704 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi196.front.sepia.ceph.com 2024-03-29T11:20:12.704 DEBUG:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-03-29T11:20:12.780 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:20:12.780 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:20:12.780 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:20:12.780 INFO:teuthology.orchestra.run.smithi196.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:20:12.780 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:20:12.780 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:20:12.780 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:20:12.780 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:20:12.780 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:20:12.780 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:20:13.270 INFO:teuthology.orchestra.run.smithi196.stdout:Discarding blocks...Done. 2024-03-29T11:20:13.273 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2024-03-29T11:20:13.273 DEBUG:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-03-29T11:20:13.415 DEBUG:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-03-29T11:20:13.448 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-03-29T11:20:13.516 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-03-29T11:20:13.516 INFO:tasks.ceph:role: osd.7 2024-03-29T11:20:13.516 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi196.front.sepia.ceph.com 2024-03-29T11:20:13.516 DEBUG:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-03-29T11:20:13.590 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-03-29T11:20:13.591 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2024-03-29T11:20:13.591 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-03-29T11:20:13.591 INFO:teuthology.orchestra.run.smithi196.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-03-29T11:20:13.591 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-03-29T11:20:13.591 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2024-03-29T11:20:13.591 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-03-29T11:20:13.591 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-03-29T11:20:13.591 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-03-29T11:20:13.591 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-03-29T11:20:14.104 INFO:teuthology.orchestra.run.smithi196.stdout:Discarding blocks...Done. 2024-03-29T11:20:14.106 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2024-03-29T11:20:14.106 DEBUG:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-03-29T11:20:14.252 DEBUG:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-03-29T11:20:14.285 DEBUG:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:20:14.402 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:14.425+0000 7fe7448bf640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-03-29T11:20:14.402 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:14.425+0000 7fe7448bf640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-03-29T11:20:14.402 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:14.425+0000 7fe7448bf640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:20:14.402 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:14.425+0000 7fe7448bf640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:20:14.403 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:14.425+0000 7fe7448bf640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-03-29T11:20:14.403 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:14.426+0000 7fe7448bf640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-03-29T11:20:17.251 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-03-29T11:20:17.280 DEBUG:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:20:17.368 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:17.390+0000 7f7eb098c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-03-29T11:20:17.368 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:17.391+0000 7f7eb098c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-03-29T11:20:17.368 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:17.391+0000 7f7eb098c640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:20:17.368 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:17.391+0000 7f7eb098c640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:20:17.368 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:17.391+0000 7f7eb098c640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-03-29T11:20:17.368 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:17.391+0000 7f7eb098c640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-03-29T11:20:20.206 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-03-29T11:20:20.235 DEBUG:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:20:20.323 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:20.346+0000 7fef8ff6b640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-03-29T11:20:20.324 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:20.347+0000 7fef8ff6b640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-03-29T11:20:20.324 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:20.347+0000 7fef8ff6b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:20:20.324 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:20.347+0000 7fef8ff6b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:20:20.324 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:20.347+0000 7fef8ff6b640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-03-29T11:20:20.324 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:20.347+0000 7fef8ff6b640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-03-29T11:20:23.186 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-03-29T11:20:23.213 DEBUG:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:20:23.301 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:23.324+0000 7f276120e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-03-29T11:20:23.301 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:23.324+0000 7f276120e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-03-29T11:20:23.301 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:23.324+0000 7f276120e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:20:23.301 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:23.324+0000 7f276120e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:20:23.301 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:23.324+0000 7f276120e640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-03-29T11:20:23.302 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:23.325+0000 7f276120e640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-03-29T11:20:26.158 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-03-29T11:20:26.190 INFO:tasks.ceph:Reading keys from all nodes... 2024-03-29T11:20:26.190 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T11:20:26.190 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-03-29T11:20:26.222 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T11:20:26.222 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-03-29T11:20:26.293 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T11:20:26.293 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-03-29T11:20:26.363 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T11:20:26.363 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-03-29T11:20:26.432 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T11:20:26.432 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-03-29T11:20:26.501 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T11:20:26.501 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-03-29T11:20:26.533 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T11:20:26.533 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-03-29T11:20:26.604 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T11:20:26.604 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-03-29T11:20:26.674 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T11:20:26.674 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-03-29T11:20:26.743 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T11:20:26.743 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-03-29T11:20:26.812 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T11:20:26.812 DEBUG:teuthology.orchestra.run.smithi086:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-03-29T11:20:26.829 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T11:20:26.829 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/etc/ceph/ceph.client.1.keyring of=/dev/stdout 2024-03-29T11:20:26.867 INFO:tasks.ceph:Adding keys to all mons... 2024-03-29T11:20:26.867 DEBUG:teuthology.orchestra.run.smithi086:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:20:26.872 DEBUG:teuthology.orchestra.run.smithi196:> sudo tee -a /etc/ceph/ceph.keyring 2024-03-29T11:20:26.911 INFO:teuthology.orchestra.run.smithi086.stdout:[mgr.y] 2024-03-29T11:20:26.911 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDbowZmjC7GNBAAcOefxi64Y7XICpOH875wSA== 2024-03-29T11:20:26.911 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.0] 2024-03-29T11:20:26.911 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDfowZmCQlsExAAVGXWvHS7RaE8rWySnFfxIQ== 2024-03-29T11:20:26.911 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.1] 2024-03-29T11:20:26.911 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDiowZmVr6MEhAA0YUSCWSKHf5L82zs9IwbzQ== 2024-03-29T11:20:26.911 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.2] 2024-03-29T11:20:26.911 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDlowZmJreFEBAAjF9UegslCgDDwdOGGoeWiw== 2024-03-29T11:20:26.911 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.3] 2024-03-29T11:20:26.911 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDoowZmbzctDxAASmnAWn4hwPXaiDcZFMJLOg== 2024-03-29T11:20:26.912 INFO:teuthology.orchestra.run.smithi086.stdout:[mgr.x] 2024-03-29T11:20:26.912 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDbowZmZP4kORAABLDdANb8V+bLohiW+PEMRQ== 2024-03-29T11:20:26.912 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.4] 2024-03-29T11:20:26.912 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDuowZm6HxpGRAA+m0Au/XTuoyN/cWmXdt9Dg== 2024-03-29T11:20:26.912 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.5] 2024-03-29T11:20:26.912 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDxowZmFpNdFxAAVMm02ub9HZzoLGcwzNgE9w== 2024-03-29T11:20:26.912 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.6] 2024-03-29T11:20:26.912 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQD0owZmizi8FBAA3Q9ZrJkF4JAv/sRH8HJuiw== 2024-03-29T11:20:26.912 INFO:teuthology.orchestra.run.smithi086.stdout:[osd.7] 2024-03-29T11:20:26.912 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQD3owZm59ZkExAAGF+fziRMw/sdi+fQ+ymwVw== 2024-03-29T11:20:26.912 INFO:teuthology.orchestra.run.smithi086.stdout:[client.0] 2024-03-29T11:20:26.912 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDcowZmDVAwABAA6JWmxUU4tqf/f7Ao2DAj3w== 2024-03-29T11:20:26.912 INFO:teuthology.orchestra.run.smithi086.stdout:[client.1] 2024-03-29T11:20:26.912 INFO:teuthology.orchestra.run.smithi086.stdout: key = AQDcowZmjPypAxAAeeAf0bt7sz8li/BAZiDXKA== 2024-03-29T11:20:26.933 INFO:teuthology.orchestra.run.smithi196.stdout:[mgr.y] 2024-03-29T11:20:26.933 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDbowZmjC7GNBAAcOefxi64Y7XICpOH875wSA== 2024-03-29T11:20:26.934 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.0] 2024-03-29T11:20:26.934 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDfowZmCQlsExAAVGXWvHS7RaE8rWySnFfxIQ== 2024-03-29T11:20:26.934 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.1] 2024-03-29T11:20:26.934 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDiowZmVr6MEhAA0YUSCWSKHf5L82zs9IwbzQ== 2024-03-29T11:20:26.934 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.2] 2024-03-29T11:20:26.934 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDlowZmJreFEBAAjF9UegslCgDDwdOGGoeWiw== 2024-03-29T11:20:26.934 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.3] 2024-03-29T11:20:26.934 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDoowZmbzctDxAASmnAWn4hwPXaiDcZFMJLOg== 2024-03-29T11:20:26.934 INFO:teuthology.orchestra.run.smithi196.stdout:[mgr.x] 2024-03-29T11:20:26.934 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDbowZmZP4kORAABLDdANb8V+bLohiW+PEMRQ== 2024-03-29T11:20:26.934 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.4] 2024-03-29T11:20:26.934 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDuowZm6HxpGRAA+m0Au/XTuoyN/cWmXdt9Dg== 2024-03-29T11:20:26.934 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.5] 2024-03-29T11:20:26.935 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDxowZmFpNdFxAAVMm02ub9HZzoLGcwzNgE9w== 2024-03-29T11:20:26.935 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.6] 2024-03-29T11:20:26.935 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQD0owZmizi8FBAA3Q9ZrJkF4JAv/sRH8HJuiw== 2024-03-29T11:20:26.935 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.7] 2024-03-29T11:20:26.935 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQD3owZm59ZkExAAGF+fziRMw/sdi+fQ+ymwVw== 2024-03-29T11:20:26.935 INFO:teuthology.orchestra.run.smithi196.stdout:[client.0] 2024-03-29T11:20:26.935 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDcowZmDVAwABAA6JWmxUU4tqf/f7Ao2DAj3w== 2024-03-29T11:20:26.935 INFO:teuthology.orchestra.run.smithi196.stdout:[client.1] 2024-03-29T11:20:26.935 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQDcowZmjPypAxAAeeAf0bt7sz8li/BAZiDXKA== 2024-03-29T11:20:26.936 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:20:26.956 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:20:27.029 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:20:27.031 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:20:27.083 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:20:27.086 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:20:27.137 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:20:27.139 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:20:27.190 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:20:27.193 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:20:27.244 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:20:27.247 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-03-29T11:20:27.298 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:20:27.300 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:20:27.352 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:20:27.354 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:20:27.406 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:20:27.408 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:20:27.460 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:20:27.462 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-03-29T11:20:27.515 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:20:27.517 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:20:27.572 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:20:27.574 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-03-29T11:20:27.628 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-03-29T11:20:27.629 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-03-29T11:20:27.655 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:20:27.753 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-03-29T11:20:27.780 DEBUG:teuthology.orchestra.run.smithi086:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-03-29T11:20:27.847 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:20:27.945 DEBUG:teuthology.orchestra.run.smithi086:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-03-29T11:20:27.971 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-03-29T11:20:27.999 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-03-29T11:20:28.097 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-03-29T11:20:28.123 DEBUG:teuthology.orchestra.run.smithi086:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:20:28.126 DEBUG:teuthology.orchestra.run.smithi196:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-03-29T11:20:28.178 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-03-29T11:20:28.178 INFO:tasks.ceph.mon.a:Restarting daemon 2024-03-29T11:20:28.178 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-03-29T11:20:28.181 INFO:tasks.ceph.mon.a:Started 2024-03-29T11:20:28.181 INFO:tasks.ceph.mon.c:Restarting daemon 2024-03-29T11:20:28.181 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-03-29T11:20:28.183 INFO:tasks.ceph.mon.c:Started 2024-03-29T11:20:28.183 INFO:tasks.ceph.mon.b:Restarting daemon 2024-03-29T11:20:28.183 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-03-29T11:20:28.221 INFO:tasks.ceph.mon.b:Started 2024-03-29T11:20:28.221 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-03-29T11:20:28.221 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-03-29T11:20:28.221 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-03-29T11:20:28.223 INFO:tasks.ceph.mgr.y:Started 2024-03-29T11:20:28.224 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-03-29T11:20:28.224 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-03-29T11:20:28.225 INFO:tasks.ceph.mgr.x:Started 2024-03-29T11:20:28.226 INFO:tasks.ceph:Setting crush tunables to default 2024-03-29T11:20:28.226 DEBUG:teuthology.orchestra.run.smithi086:> sudo ceph --cluster ceph osd crush tunables default 2024-03-29T11:20:28.328 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:28.351+0000 7faf0538d640 10 monclient: get_monmap_and_config 2024-03-29T11:20:28.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:28.351+0000 7faf0538d640 10 monclient: build_initial_monmap 2024-03-29T11:20:28.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:28.351+0000 7faf0538d640 10 monclient: monmap: 2024-03-29T11:20:28.329 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-29T11:20:28.329 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:28.329 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T11:20:28.352919+0000 2024-03-29T11:20:28.329 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T11:20:28.352919+0000 2024-03-29T11:20:28.329 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:28.329 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:28.329 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:20:28.329 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:20:28.329 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:20:28.329 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T11:20:28.329 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:28.352+0000 7faf0538d640 1 Processor -- start 2024-03-29T11:20:28.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:28.352+0000 7faf0538d640 1 -- start start 2024-03-29T11:20:28.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:28.352+0000 7faf0538d640 10 monclient: init 2024-03-29T11:20:28.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:28.353+0000 7faf0538d640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:28.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:28.353+0000 7faf0538d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:28.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:28.353+0000 7faf0538d640 10 monclient(hunting): picked mon.noname-b con 0x7faf0009ac40 addr v1:172.21.15.86:6790/0 2024-03-29T11:20:28.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:28.353+0000 7faf0538d640 10 monclient(hunting): picked mon.noname-a con 0x7faf0009c870 addr v1:172.21.15.86:6789/0 2024-03-29T11:20:28.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:28.353+0000 7faf0538d640 10 monclient(hunting): picked mon.noname-c con 0x7faf000b4360 addr v1:172.21.15.196:6789/0 2024-03-29T11:20:28.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:28.353+0000 7faf0538d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf0009dec0 con 0x7faf0009c870 2024-03-29T11:20:28.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:28.353+0000 7faf0538d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf0009da30 con 0x7faf000b4360 2024-03-29T11:20:28.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:28.353+0000 7faf0538d640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf000b39c0 con 0x7faf0009ac40 2024-03-29T11:20:28.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:28.353+0000 7faf0538d640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:28.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:28.353+0000 7faf0538d640 10 monclient(hunting): authenticate will time out at 852.671291s 2024-03-29T11:20:28.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:28.353+0000 7faefe7fc640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7faf0009c870 0x7faf0013f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:54400/0 (socket says 172.21.15.86:54400) 2024-03-29T11:20:28.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:28.353+0000 7faefe7fc640 1 -- 172.21.15.86:0/2787446319 learned_addr learned my addr 172.21.15.86:0/2787446319 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:20:28.332 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:28.353+0000 7faefeffd640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7faf0009ac40 0x7faf000b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:47880/0 (socket says 172.21.15.86:47880) 2024-03-29T11:20:31.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:31.353+0000 7faefcff9640 10 monclient(hunting): tick 2024-03-29T11:20:31.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:31.353+0000 7faefcff9640 10 monclient(hunting): _check_auth_tickets 2024-03-29T11:20:31.330 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:31.353+0000 7faefcff9640 1 monclient(hunting): continuing hunt 2024-03-29T11:20:31.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:31.353+0000 7faefcff9640 10 monclient(hunting): _reopen_session rank -1 2024-03-29T11:20:31.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:31.353+0000 7faefcff9640 1 -- 172.21.15.86:0/2787446319 >> v1:172.21.15.86:6790/0 conn(0x7faf0009ac40 legacy=0x7faf000b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:31.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:31.353+0000 7faefcff9640 1 -- 172.21.15.86:0/2787446319 >> v1:172.21.15.196:6789/0 conn(0x7faf000b4360 legacy=0x7faf0014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:31.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:31.353+0000 7faefcff9640 1 -- 172.21.15.86:0/2787446319 >> v1:172.21.15.86:6789/0 conn(0x7faf0009c870 legacy=0x7faf0013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:31.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:31.353+0000 7faefcff9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:31.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:31.353+0000 7faefcff9640 10 monclient(hunting): picked mon.noname-b con 0x7faeec0036e0 addr v1:172.21.15.86:6790/0 2024-03-29T11:20:31.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:31.353+0000 7faefcff9640 10 monclient(hunting): picked mon.noname-c con 0x7faeec0072d0 addr v1:172.21.15.196:6789/0 2024-03-29T11:20:31.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:31.353+0000 7faefcff9640 10 monclient(hunting): picked mon.noname-a con 0x7faeec00b020 addr v1:172.21.15.86:6789/0 2024-03-29T11:20:31.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:31.353+0000 7faefcff9640 1 -- 172.21.15.86:0/2787446319 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf000b5160 con 0x7faeec00b020 2024-03-29T11:20:31.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:31.353+0000 7faefcff9640 1 -- 172.21.15.86:0/2787446319 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeec00fdf0 con 0x7faeec0072d0 2024-03-29T11:20:31.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:31.353+0000 7faefcff9640 1 -- 172.21.15.86:0/2787446319 --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faeec011030 con 0x7faeec0036e0 2024-03-29T11:20:31.331 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:31.353+0000 7faefcff9640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:33.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.520+0000 7faefdffb640 1 -- 172.21.15.86:0/2787446319 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2457087580 0 0) 0x7faf000b5160 con 0x7faeec00b020 2024-03-29T11:20:33.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.520+0000 7faefdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:33.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.520+0000 7faefdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:33.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.520+0000 7faefdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:33.497 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.520+0000 7faefdffb640 10 monclient(hunting): my global_id is 4101 2024-03-29T11:20:33.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.520+0000 7faefdffb640 1 -- 172.21.15.86:0/2787446319 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faee8003520 con 0x7faeec00b020 2024-03-29T11:20:33.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.520+0000 7faefe7fc640 1 -- 172.21.15.86:0/2787446319 >> v1:172.21.15.196:6789/0 conn(0x7faeec0072d0 legacy=0x7faeec009760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-03-29T11:20:33.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.520+0000 7faefe7fc640 1 -- 172.21.15.86:0/2787446319 >> v1:172.21.15.196:6789/0 conn(0x7faeec0072d0 legacy=0x7faeec009760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:20:33.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.520+0000 7faefe7fc640 1 --1- 172.21.15.86:0/2787446319 >> v1:172.21.15.196:6789/0 conn(0x7faeec0072d0 0x7faeec009760 :-1 s=OPENED pgs=11 cs=1 l=1).handle_message read tag failed 2024-03-29T11:20:33.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.520+0000 7faefe7fc640 1 --1- 172.21.15.86:0/2787446319 >> v1:172.21.15.196:6789/0 conn(0x7faeec0072d0 0x7faeec009760 :-1 s=OPENED pgs=11 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:20:33.498 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.520+0000 7faefdffb640 10 monclient(hunting): ms_handle_reset hunted mon v1:172.21.15.196:6789/0 2024-03-29T11:20:33.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.522+0000 7faefdffb640 1 -- 172.21.15.86:0/2787446319 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1084584289 0 0) 0x7faee8003520 con 0x7faeec00b020 2024-03-29T11:20:33.499 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.522+0000 7faefdffb640 1 -- 172.21.15.86:0/2787446319 >> v1:172.21.15.86:6790/0 conn(0x7faeec0036e0 legacy=0x7faeec005b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:33.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.522+0000 7faefdffb640 1 -- 172.21.15.86:0/2787446319 >> v1:172.21.15.196:6789/0 conn(0x7faeec0072d0 legacy=0x7faeec009760 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:20:33.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.522+0000 7faefdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:20:33.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.522+0000 7faefdffb640 1 monclient: found mon.noname-a 2024-03-29T11:20:33.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.522+0000 7faefdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:33.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.522+0000 7faefdffb640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.86:6789/0 2024-03-29T11:20:33.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.522+0000 7faefdffb640 1 -- 172.21.15.86:0/2787446319 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faeec012290 con 0x7faeec00b020 2024-03-29T11:20:33.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.522+0000 7faefdffb640 10 monclient: _finish_auth 0 2024-03-29T11:20:33.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.522+0000 7faefdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:20:33.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.522+0000 7faefdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:33.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.522+0000 7faefdffb640 1 -- 172.21.15.86:0/2787446319 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7faef8001a50 con 0x7faeec00b020 2024-03-29T11:20:33.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.522+0000 7faefdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:33.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.522+0000 7faefdffb640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:33.500 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.522+0000 7faefdffb640 10 monclient: dump: 2024-03-29T11:20:33.501 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T11:20:33.501 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:33.501 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:33.501 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:33.501 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:33.501 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:33.501 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:33.501 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:33.501 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:33.501 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T11:20:33.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.522+0000 7faf0538d640 5 monclient: authenticate success, global_id 4101 2024-03-29T11:20:33.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.522+0000 7faf0538d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:33.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.522+0000 7faefdffb640 1 -- 172.21.15.86:0/2787446319 <== mon.0 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faef8001c20 con 0x7faeec00b020 2024-03-29T11:20:33.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.522+0000 7faefdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:33.501 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.522+0000 7faefdffb640 1 -- 172.21.15.86:0/2787446319 <== mon.0 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7faef8001ff0 con 0x7faeec00b020 2024-03-29T11:20:33.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.522+0000 7faf0538d640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:33.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.522+0000 7faf0538d640 4 set_mon_vals no callback set 2024-03-29T11:20:33.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.522+0000 7faefdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:33.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.522+0000 7faefdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:33.502 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.522+0000 7faefdffb640 10 monclient: dump: 2024-03-29T11:20:33.502 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T11:20:33.502 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:33.502 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:33.502 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:33.502 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:33.502 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:33.502 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:33.503 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:33.503 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:33.503 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T11:20:33.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.523+0000 7faf0538d640 10 monclient: shutdown 2024-03-29T11:20:33.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.523+0000 7faf0538d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:33.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.523+0000 7faf0538d640 1 -- 172.21.15.86:0/2787446319 >> v1:172.21.15.86:6789/0 conn(0x7faeec00b020 legacy=0x7faeec00d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:33.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.523+0000 7faf0538d640 1 -- 172.21.15.86:0/2787446319 shutdown_connections 2024-03-29T11:20:33.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.523+0000 7faf0538d640 1 -- 172.21.15.86:0/2787446319 >> 172.21.15.86:0/2787446319 conn(0x7faf00066b00 msgr2=0x7faf00066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:33.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.524+0000 7faf0538d640 1 -- 172.21.15.86:0/2787446319 shutdown_connections 2024-03-29T11:20:33.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.524+0000 7faf0538d640 1 -- 172.21.15.86:0/2787446319 wait complete. 2024-03-29T11:20:33.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.524+0000 7faf0538d640 10 monclient: build_initial_monmap 2024-03-29T11:20:33.508 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.524+0000 7faf0538d640 10 monclient: monmap: 2024-03-29T11:20:33.508 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-29T11:20:33.508 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:33.509 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-03-29T11:20:33.509 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-03-29T11:20:33.509 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:33.509 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:33.509 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:20:33.509 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:20:33.509 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:20:33.509 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T11:20:33.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.525+0000 7faf0538d640 1 librados: starting msgr at 2024-03-29T11:20:33.509 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.525+0000 7faf0538d640 1 librados: starting objecter 2024-03-29T11:20:33.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.525+0000 7faf0538d640 1 Processor -- start 2024-03-29T11:20:33.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.525+0000 7faf0538d640 1 -- start start 2024-03-29T11:20:33.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.525+0000 7faf0538d640 1 librados: setting wanted keys 2024-03-29T11:20:33.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.525+0000 7faf0538d640 1 librados: calling monclient init 2024-03-29T11:20:33.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.525+0000 7faf0538d640 10 monclient: init 2024-03-29T11:20:33.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.526+0000 7faf0538d640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:33.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.526+0000 7faf0538d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:33.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.526+0000 7faf0538d640 10 monclient(hunting): picked mon.noname-c con 0x7faeec0036e0 addr v1:172.21.15.86:6790/0 2024-03-29T11:20:33.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.526+0000 7faf0538d640 10 monclient(hunting): picked mon.noname-a con 0x7faeec0072d0 addr v1:172.21.15.86:6789/0 2024-03-29T11:20:33.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.526+0000 7faf0538d640 10 monclient(hunting): picked mon.noname-b con 0x7faeec00b020 addr v1:172.21.15.196:6789/0 2024-03-29T11:20:33.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.526+0000 7faf0538d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf001d7060 con 0x7faeec0072d0 2024-03-29T11:20:33.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.526+0000 7faefeffd640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7faeec0036e0 0x7faf000adad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:60384/0 (socket says 172.21.15.86:60384) 2024-03-29T11:20:33.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.526+0000 7faefeffd640 1 -- 172.21.15.86:0/3083190456 learned_addr learned my addr 172.21.15.86:0/3083190456 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:20:33.510 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.526+0000 7faf0538d640 1 -- 172.21.15.86:0/3083190456 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf000af480 con 0x7faeec00b020 2024-03-29T11:20:33.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.526+0000 7faf0538d640 1 -- 172.21.15.86:0/3083190456 --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faf000ae1e0 con 0x7faeec0036e0 2024-03-29T11:20:33.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.526+0000 7faf0538d640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:33.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.526+0000 7faf0538d640 10 monclient(hunting): authenticate will time out at 857.844717s 2024-03-29T11:20:33.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.526+0000 7faee77fe640 10 client.?.objecter ms_handle_connect 0x7faeec0036e0 2024-03-29T11:20:33.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.526+0000 7faee77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:33.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.526+0000 7faee77fe640 10 client.?.objecter ms_handle_connect 0x7faeec0072d0 2024-03-29T11:20:33.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.526+0000 7faee77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:33.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.526+0000 7faee77fe640 1 -- 172.21.15.86:0/3083190456 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2751909614 0 0) 0x7faf001d7060 con 0x7faeec0072d0 2024-03-29T11:20:33.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.526+0000 7faee77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:33.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.526+0000 7faee77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:33.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.526+0000 7faee77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:33.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.526+0000 7faee77fe640 10 monclient(hunting): my global_id is 4110 2024-03-29T11:20:33.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.526+0000 7faee77fe640 1 -- 172.21.15.86:0/3083190456 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faeec010740 con 0x7faeec0072d0 2024-03-29T11:20:33.511 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.527+0000 7faee77fe640 10 client.?.objecter ms_handle_connect 0x7faeec00b020 2024-03-29T11:20:33.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.527+0000 7faee77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:33.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.528+0000 7faee77fe640 1 -- 172.21.15.86:0/3083190456 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 340172468 0 0) 0x7faf000ae1e0 con 0x7faeec0036e0 2024-03-29T11:20:33.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.528+0000 7faee77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:33.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.528+0000 7faee77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:33.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.528+0000 7faee77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:33.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.528+0000 7faee77fe640 10 monclient(hunting): my global_id is 4103 2024-03-29T11:20:33.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.528+0000 7faee77fe640 1 -- 172.21.15.86:0/3083190456 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf001d7060 con 0x7faeec0036e0 2024-03-29T11:20:33.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.528+0000 7faee77fe640 1 -- 172.21.15.86:0/3083190456 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2820271927 0 0) 0x7faf001d7060 con 0x7faeec0036e0 2024-03-29T11:20:33.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.528+0000 7faee77fe640 1 -- 172.21.15.86:0/3083190456 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf000ae1e0 con 0x7faeec0036e0 2024-03-29T11:20:33.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.528+0000 7faee77fe640 1 -- 172.21.15.86:0/3083190456 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7faef00017e0 con 0x7faeec0036e0 2024-03-29T11:20:33.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.528+0000 7faee77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:33.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.528+0000 7faee77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:33.512 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.528+0000 7faee77fe640 10 monclient(hunting): dump: 2024-03-29T11:20:33.512 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T11:20:33.512 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:33.513 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:33.513 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:33.513 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:33.513 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:33.513 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:33.513 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:33.513 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:33.513 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T11:20:33.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.528+0000 7faee77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:33.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.528+0000 7faee77fe640 1 -- 172.21.15.86:0/3083190456 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1637658562 0 0) 0x7faeec010740 con 0x7faeec0072d0 2024-03-29T11:20:33.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.528+0000 7faee77fe640 1 -- 172.21.15.86:0/3083190456 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7faf001d7060 con 0x7faeec0072d0 2024-03-29T11:20:33.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.528+0000 7faee77fe640 1 -- 172.21.15.86:0/3083190456 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7faef40017d0 con 0x7faeec0072d0 2024-03-29T11:20:33.513 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.528+0000 7faee77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:33.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.528+0000 7faee77fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:33.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.528+0000 7faee77fe640 10 monclient(hunting): dump: 2024-03-29T11:20:33.514 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T11:20:33.514 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:33.514 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:33.514 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:33.514 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:33.514 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:33.514 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:33.514 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:33.514 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:33.514 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T11:20:33.514 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.529+0000 7faee77fe640 1 -- 172.21.15.86:0/3083190456 <== mon.0 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2764492523 0 0) 0x7faf001d7060 con 0x7faeec0072d0 2024-03-29T11:20:33.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.529+0000 7faee77fe640 1 -- 172.21.15.86:0/3083190456 >> v1:172.21.15.86:6790/0 conn(0x7faeec0036e0 legacy=0x7faf000adad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:33.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.529+0000 7faee77fe640 1 -- 172.21.15.86:0/3083190456 >> v1:172.21.15.196:6789/0 conn(0x7faeec00b020 legacy=0x7faf00136cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:33.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.529+0000 7faee77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:20:33.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.529+0000 7faee77fe640 1 monclient: found mon.a 2024-03-29T11:20:33.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.529+0000 7faee77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:33.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.529+0000 7faee77fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:20:33.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.529+0000 7faee77fe640 1 -- 172.21.15.86:0/3083190456 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf000b6410 con 0x7faeec0072d0 2024-03-29T11:20:33.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.529+0000 7faee77fe640 10 monclient: _finish_auth 0 2024-03-29T11:20:33.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.529+0000 7faee77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:20:33.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.529+0000 7faee77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:33.515 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.529+0000 7faf0538d640 5 monclient: authenticate success, global_id 4110 2024-03-29T11:20:33.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.529+0000 7faf0538d640 10 monclient: _renew_subs 2024-03-29T11:20:33.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.529+0000 7faf0538d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:20:33.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.529+0000 7faf0538d640 1 -- 172.21.15.86:0/3083190456 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf001d7c00 con 0x7faeec0072d0 2024-03-29T11:20:33.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.529+0000 7faf0538d640 10 client.4110.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:33.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.529+0000 7faf0538d640 10 monclient: _renew_subs 2024-03-29T11:20:33.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.529+0000 7faf0538d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:20:33.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.529+0000 7faf0538d640 1 -- 172.21.15.86:0/3083190456 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7faf000ad270 con 0x7faeec0072d0 2024-03-29T11:20:33.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.529+0000 7faf0538d640 1 librados: init done 2024-03-29T11:20:33.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.529+0000 7faee77fe640 1 -- 172.21.15.86:0/3083190456 <== mon.0 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7faef4001cc0 con 0x7faeec0072d0 2024-03-29T11:20:33.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.529+0000 7faee77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:33.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.529+0000 7faee77fe640 1 -- 172.21.15.86:0/3083190456 <== mon.0 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7faef4004980 con 0x7faeec0072d0 2024-03-29T11:20:33.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.529+0000 7faee77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:33.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.529+0000 7faee77fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:33.516 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.529+0000 7faee77fe640 10 monclient: dump: 2024-03-29T11:20:33.517 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T11:20:33.517 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:33.517 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:33.517 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:33.517 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:33.517 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:33.517 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:33.517 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:33.517 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:33.517 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T11:20:33.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.529+0000 7faee77fe640 1 -- 172.21.15.86:0/3083190456 <== mon.0 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7faef4004eb0 con 0x7faeec0072d0 2024-03-29T11:20:33.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.529+0000 7faefd7fa640 4 set_mon_vals no callback set 2024-03-29T11:20:33.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.529+0000 7faee77fe640 1 -- 172.21.15.86:0/3083190456 <== mon.0 v1:172.21.15.86:6789/0 8 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (unknown 3736557734 0 0) 0x7faef4006460 con 0x7faeec0072d0 2024-03-29T11:20:33.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.529+0000 7faee77fe640 10 client.4110.objecter ms_dispatch 0x7faf00000b90 osd_map(1..1 src has 1..1) v4 2024-03-29T11:20:33.517 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.529+0000 7faee77fe640 3 client.4110.objecter handle_osd_map got epochs [1,1] > 0 2024-03-29T11:20:33.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.529+0000 7faee77fe640 3 client.4110.objecter handle_osd_map decoding full epoch 1 2024-03-29T11:20:33.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.529+0000 7faee77fe640 20 client.4110.objecter dump_active .. 0 homeless 2024-03-29T11:20:33.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.530+0000 7faf0538d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:33.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.530+0000 7faf0538d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:33.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.530+0000 7faf0538d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:20:33.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.530+0000 7faf0538d640 1 -- 172.21.15.86:0/3083190456 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faee0005080 con 0x7faeec0072d0 2024-03-29T11:20:33.518 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.532+0000 7faee77fe640 1 -- 172.21.15.86:0/3083190456 <== mon.0 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7faef4006620 con 0x7faeec0072d0 2024-03-29T11:20:33.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.532+0000 7faee77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:33.519 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.532+0000 7faee77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:33.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.621+0000 7faf0538d640 10 start_mon_command cmd=[{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:20:33.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.621+0000 7faf0538d640 10 monclient: _send_command 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:20:33.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.621+0000 7faf0538d640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:20:33.598 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.621+0000 7faf0538d640 1 -- 172.21.15.86:0/3083190456 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7faee0005330 con 0x7faeec0072d0 2024-03-29T11:20:33.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.625+0000 7faee77fe640 1 -- 172.21.15.86:0/3083190456 <== mon.0 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (unknown 2277712955 0 0) 0x7faef4005240 con 0x7faeec0072d0 2024-03-29T11:20:33.601 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.625+0000 7faee77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd crush tunables", "profile": "default"}] 2024-03-29T11:20:33.602 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.625+0000 7faee77fe640 10 monclient: _finish_command 2 = system:0 adjusted tunables profile to default 2024-03-29T11:20:33.602 INFO:teuthology.orchestra.run.smithi086.stderr:adjusted tunables profile to default 2024-03-29T11:20:33.603 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.626+0000 7faf0538d640 10 librados: watch_flush enter 2024-03-29T11:20:33.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.626+0000 7faf0538d640 10 librados: watch_flush exit 2024-03-29T11:20:33.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.626+0000 7faf0538d640 20 client.4110.objecter shutdown clearing up homeless session... 2024-03-29T11:20:33.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.626+0000 7faf0538d640 10 client.4110.objecter successfully canceled tick 2024-03-29T11:20:33.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.626+0000 7faf0538d640 10 monclient: shutdown 2024-03-29T11:20:33.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.626+0000 7faf0538d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:33.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.626+0000 7faf0538d640 1 -- 172.21.15.86:0/3083190456 >> v1:172.21.15.86:6789/0 conn(0x7faeec0072d0 legacy=0x7faf001d74f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:33.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.626+0000 7faf0538d640 1 -- 172.21.15.86:0/3083190456 shutdown_connections 2024-03-29T11:20:33.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.626+0000 7faf0538d640 1 -- 172.21.15.86:0/3083190456 >> 172.21.15.86:0/3083190456 conn(0x7faf00066b00 msgr2=0x7faf000b2550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:33.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.626+0000 7faf0538d640 1 -- 172.21.15.86:0/3083190456 shutdown_connections 2024-03-29T11:20:33.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.626+0000 7faf0538d640 1 -- 172.21.15.86:0/3083190456 wait complete. 2024-03-29T11:20:33.604 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:33.626+0000 7faf0538d640 1 librados: shutdown 2024-03-29T11:20:33.614 INFO:tasks.ceph:check_enable_crimson: False 2024-03-29T11:20:33.614 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-03-29T11:20:33.614 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T11:20:33.615 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-03-29T11:20:33.641 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T11:20:33.642 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-03-29T11:20:33.708 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T11:20:33.708 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-03-29T11:20:33.775 DEBUG:teuthology.orchestra.run.smithi086:> set -ex 2024-03-29T11:20:33.775 DEBUG:teuthology.orchestra.run.smithi086:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-03-29T11:20:33.843 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T11:20:33.843 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-03-29T11:20:33.870 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T11:20:33.870 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-03-29T11:20:33.937 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T11:20:33.938 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-03-29T11:20:33.968 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T11:20:33.992+0000 7fee3a432040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:20:33.995 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-03-29T11:20:33.995 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-03-29T11:20:33.997 INFO:tasks.ceph.mgr.y.smithi086.stderr:2024-03-29T11:20:34.021+0000 7f4bd1e42040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-29T11:20:34.026 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T11:20:34.049+0000 7fee3a432040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:20:34.051 INFO:tasks.ceph.mgr.y.smithi086.stderr:2024-03-29T11:20:34.075+0000 7f4bd1e42040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-29T11:20:34.053 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 69e25ec4-1959-46d9-8d55-eb4e52eaf6f1 0 2024-03-29T11:20:34.193 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.216+0000 7f3128e25640 10 monclient: get_monmap_and_config 2024-03-29T11:20:34.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.216+0000 7f3128e25640 10 monclient: build_initial_monmap 2024-03-29T11:20:34.194 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.216+0000 7f3128e25640 10 monclient: monmap: 2024-03-29T11:20:34.194 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-29T11:20:34.194 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:34.194 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:20:34.217684+0000 2024-03-29T11:20:34.194 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:20:34.217684+0000 2024-03-29T11:20:34.194 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:34.194 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:34.194 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:20:34.194 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:20:34.195 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:20:34.195 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:34.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.217+0000 7f3128e25640 1 Processor -- start 2024-03-29T11:20:34.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.217+0000 7f3128e25640 1 -- start start 2024-03-29T11:20:34.195 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.217+0000 7f3128e25640 10 monclient: init 2024-03-29T11:20:34.206 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.217+0000 7f3128e25640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:34.206 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.217+0000 7f3128e25640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:34.206 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.217+0000 7f3128e25640 10 monclient(hunting): picked mon.noname-c con 0x7f312409ac40 addr v1:172.21.15.196:6789/0 2024-03-29T11:20:34.206 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.217+0000 7f3128e25640 10 monclient(hunting): picked mon.noname-a con 0x7f312409c870 addr v1:172.21.15.86:6789/0 2024-03-29T11:20:34.206 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.217+0000 7f3128e25640 10 monclient(hunting): picked mon.noname-b con 0x7f31240b4360 addr v1:172.21.15.86:6790/0 2024-03-29T11:20:34.206 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.217+0000 7f3128e25640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f312409dec0 con 0x7f312409c870 2024-03-29T11:20:34.206 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.217+0000 7f3128e25640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f312409da30 con 0x7f312409ac40 2024-03-29T11:20:34.206 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.217+0000 7f3128e25640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31240b39c0 con 0x7f31240b4360 2024-03-29T11:20:34.206 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.217+0000 7f3128e25640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:34.206 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.217+0000 7f3128e25640 10 monclient(hunting): authenticate will time out at 879.505960s 2024-03-29T11:20:34.207 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312259c640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f312409ac40 0x7f31240b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:51480/0 (socket says 172.21.15.196:51480) 2024-03-29T11:20:34.207 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312259c640 1 -- 172.21.15.196:0/818533986 learned_addr learned my addr 172.21.15.196:0/818533986 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-29T11:20:34.207 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 1 -- 172.21.15.196:0/818533986 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3956084158 0 0) 0x7f312409da30 con 0x7f312409ac40 2024-03-29T11:20:34.207 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 10 monclient(hunting): _negotiate 2024-03-29T11:20:34.207 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:34.207 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:34.207 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 10 monclient(hunting): my global_id is 4111 2024-03-29T11:20:34.207 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 1 -- 172.21.15.196:0/818533986 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f310c003520 con 0x7f312409ac40 2024-03-29T11:20:34.207 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 1 -- 172.21.15.196:0/818533986 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2889581788 0 0) 0x7f3118001110 con 0x7f312409c870 2024-03-29T11:20:34.207 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 10 monclient(hunting): _negotiate 2024-03-29T11:20:34.207 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:34.207 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:34.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 10 monclient(hunting): my global_id is 4113 2024-03-29T11:20:34.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 1 -- 172.21.15.196:0/818533986 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f312409da30 con 0x7f312409c870 2024-03-29T11:20:34.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 1 -- 172.21.15.196:0/818533986 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2503129980 0 0) 0x7f31240b39c0 con 0x7f31240b4360 2024-03-29T11:20:34.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 10 monclient(hunting): _negotiate 2024-03-29T11:20:34.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:34.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:34.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 10 monclient(hunting): my global_id is 4106 2024-03-29T11:20:34.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 1 -- 172.21.15.196:0/818533986 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3118001110 con 0x7f31240b4360 2024-03-29T11:20:34.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 1 -- 172.21.15.196:0/818533986 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 270047372 0 0) 0x7f310c003520 con 0x7f312409ac40 2024-03-29T11:20:34.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 1 -- 172.21.15.196:0/818533986 >> v1:172.21.15.86:6790/0 conn(0x7f31240b4360 legacy=0x7f312414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 1 -- 172.21.15.196:0/818533986 >> v1:172.21.15.86:6789/0 conn(0x7f312409c870 legacy=0x7f312413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.208 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 10 monclient: _finish_hunting 0 2024-03-29T11:20:34.209 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 1 monclient: found mon.noname-c 2024-03-29T11:20:34.209 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:34.209 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T11:20:34.209 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 1 -- 172.21.15.196:0/818533986 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31240b5160 con 0x7f312409ac40 2024-03-29T11:20:34.209 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 10 monclient: _finish_auth 0 2024-03-29T11:20:34.209 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 10 monclient: _check_auth_tickets 2024-03-29T11:20:34.209 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:34.209 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 1 -- 172.21.15.196:0/818533986 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f3114001690 con 0x7f312409ac40 2024-03-29T11:20:34.209 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f3128e25640 5 monclient: authenticate success, global_id 4111 2024-03-29T11:20:34.209 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f3128e25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:34.209 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:34.209 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:34.209 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 10 monclient: dump: 2024-03-29T11:20:34.210 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:34.210 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:34.210 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.210 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.210 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:34.210 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:34.210 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:34.210 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:34.210 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:34.210 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:34.210 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f3128e25640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:34.210 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 1 -- 172.21.15.196:0/818533986 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3114001870 con 0x7f312409ac40 2024-03-29T11:20:34.211 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:34.211 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f3128e25640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:34.211 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 1 -- 172.21.15.196:0/818533986 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f3114001c80 con 0x7f312409ac40 2024-03-29T11:20:34.211 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f3128e25640 4 set_mon_vals no callback set 2024-03-29T11:20:34.211 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:34.211 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:34.211 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.218+0000 7f312159a640 10 monclient: dump: 2024-03-29T11:20:34.211 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:34.211 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:34.211 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.211 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.211 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:34.211 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:34.212 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:34.212 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:34.212 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:34.212 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:34.212 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.219+0000 7f3128e25640 10 monclient: shutdown 2024-03-29T11:20:34.212 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.219+0000 7f3128e25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:34.212 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.219+0000 7f3128e25640 1 -- 172.21.15.196:0/818533986 >> v1:172.21.15.196:6789/0 conn(0x7f312409ac40 legacy=0x7f31240b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.212 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.219+0000 7f3128e25640 1 -- 172.21.15.196:0/818533986 shutdown_connections 2024-03-29T11:20:34.212 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.219+0000 7f3128e25640 1 -- 172.21.15.196:0/818533986 >> 172.21.15.196:0/818533986 conn(0x7f3124066b00 msgr2=0x7f3124066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:34.212 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.219+0000 7f3128e25640 1 -- 172.21.15.196:0/818533986 shutdown_connections 2024-03-29T11:20:34.212 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.219+0000 7f3128e25640 1 -- 172.21.15.196:0/818533986 wait complete. 2024-03-29T11:20:34.212 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.219+0000 7f3128e25640 10 monclient: build_initial_monmap 2024-03-29T11:20:34.212 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.219+0000 7f3128e25640 10 monclient: monmap: 2024-03-29T11:20:34.213 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-29T11:20:34.213 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:34.213 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 0.000000 2024-03-29T11:20:34.213 INFO:teuthology.orchestra.run.smithi196.stderr:created 0.000000 2024-03-29T11:20:34.213 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:34.213 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:34.213 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:20:34.213 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:20:34.213 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:20:34.213 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:34.213 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.220+0000 7f3128e25640 1 librados: starting msgr at 2024-03-29T11:20:34.213 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.220+0000 7f3128e25640 1 librados: starting objecter 2024-03-29T11:20:34.214 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.220+0000 7f3128e25640 1 Processor -- start 2024-03-29T11:20:34.214 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.220+0000 7f3128e25640 1 -- start start 2024-03-29T11:20:34.214 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.220+0000 7f3128e25640 1 librados: setting wanted keys 2024-03-29T11:20:34.214 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.220+0000 7f3128e25640 1 librados: calling monclient init 2024-03-29T11:20:34.214 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.220+0000 7f3128e25640 10 monclient: init 2024-03-29T11:20:34.214 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f3128e25640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:34.214 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f3128e25640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:34.214 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f3128e25640 10 monclient(hunting): picked mon.noname-b con 0x7f312409ac40 addr v1:172.21.15.196:6789/0 2024-03-29T11:20:34.214 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f3128e25640 10 monclient(hunting): picked mon.noname-c con 0x7f31241ccb70 addr v1:172.21.15.86:6790/0 2024-03-29T11:20:34.214 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f3128e25640 10 monclient(hunting): picked mon.noname-a con 0x7f31241425f0 addr v1:172.21.15.86:6789/0 2024-03-29T11:20:34.214 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f3128e25640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31241cd520 con 0x7f31241425f0 2024-03-29T11:20:34.214 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f3128e25640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31240acd00 con 0x7f312409ac40 2024-03-29T11:20:34.215 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f3128e25640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31241cd000 con 0x7f31241ccb70 2024-03-29T11:20:34.215 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f3128e25640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:34.215 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f3128e25640 10 monclient(hunting): authenticate will time out at 879.509482s 2024-03-29T11:20:34.215 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f312259c640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f312409ac40 0x7f31241da690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:51486/0 (socket says 172.21.15.196:51486) 2024-03-29T11:20:34.215 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f312259c640 1 -- 172.21.15.196:0/3571397586 learned_addr learned my addr 172.21.15.196:0/3571397586 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-29T11:20:34.215 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f3112ffd640 10 client.?.objecter ms_handle_connect 0x7f312409ac40 2024-03-29T11:20:34.215 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f3112ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:34.215 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f3112ffd640 10 client.?.objecter ms_handle_connect 0x7f31241ccb70 2024-03-29T11:20:34.215 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f3112ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:34.215 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f3112ffd640 1 -- 172.21.15.196:0/3571397586 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1379904382 0 0) 0x7f31240acd00 con 0x7f312409ac40 2024-03-29T11:20:34.215 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f3112ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:34.215 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f3112ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:34.215 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f3112ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:34.216 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f3112ffd640 10 monclient(hunting): my global_id is 4114 2024-03-29T11:20:34.216 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f3112ffd640 1 -- 172.21.15.196:0/3571397586 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3104004eb0 con 0x7f312409ac40 2024-03-29T11:20:34.216 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f3112ffd640 10 client.?.objecter ms_handle_connect 0x7f31241425f0 2024-03-29T11:20:34.216 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f3112ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:34.216 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f3112ffd640 1 -- 172.21.15.196:0/3571397586 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1752988598 0 0) 0x7f31241cd000 con 0x7f31241ccb70 2024-03-29T11:20:34.216 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f3112ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:34.216 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f3112ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:34.216 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f3112ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:34.216 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f3112ffd640 10 monclient(hunting): my global_id is 4109 2024-03-29T11:20:34.216 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f3112ffd640 1 -- 172.21.15.196:0/3571397586 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31240acd00 con 0x7f31241ccb70 2024-03-29T11:20:34.216 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f3112ffd640 1 -- 172.21.15.196:0/3571397586 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 411417401 0 0) 0x7f31241cd520 con 0x7f31241425f0 2024-03-29T11:20:34.216 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f3112ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:34.216 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f3112ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:34.217 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f3112ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:34.217 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f3112ffd640 10 monclient(hunting): my global_id is 4116 2024-03-29T11:20:34.217 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.221+0000 7f3112ffd640 1 -- 172.21.15.196:0/3571397586 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f31241cd000 con 0x7f31241425f0 2024-03-29T11:20:34.217 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 1 -- 172.21.15.196:0/3571397586 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2545318696 0 0) 0x7f3104004eb0 con 0x7f312409ac40 2024-03-29T11:20:34.217 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 1 -- 172.21.15.196:0/3571397586 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31241cd520 con 0x7f312409ac40 2024-03-29T11:20:34.217 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 1 -- 172.21.15.196:0/3571397586 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f3114001bc0 con 0x7f312409ac40 2024-03-29T11:20:34.217 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:34.217 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:34.217 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 10 monclient(hunting): dump: 2024-03-29T11:20:34.217 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:34.217 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:34.217 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.218 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.218 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:34.218 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:34.218 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:34.218 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:34.218 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:34.218 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:34.218 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:34.218 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 1 -- 172.21.15.196:0/3571397586 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2262483418 0 0) 0x7f31240acd00 con 0x7f31241ccb70 2024-03-29T11:20:34.218 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 1 -- 172.21.15.196:0/3571397586 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3104004eb0 con 0x7f31241ccb70 2024-03-29T11:20:34.218 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 1 -- 172.21.15.196:0/3571397586 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f31180018a0 con 0x7f31241ccb70 2024-03-29T11:20:34.218 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:34.218 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:34.219 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 10 monclient(hunting): dump: 2024-03-29T11:20:34.219 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:34.219 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:34.219 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.219 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.219 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:34.219 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:34.219 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:34.219 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:34.219 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:34.219 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:34.219 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 1 -- 172.21.15.196:0/3571397586 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2543544846 0 0) 0x7f31241cd000 con 0x7f31241425f0 2024-03-29T11:20:34.219 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 1 -- 172.21.15.196:0/3571397586 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f31240acd00 con 0x7f31241425f0 2024-03-29T11:20:34.220 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 1 -- 172.21.15.196:0/3571397586 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f311c001b50 con 0x7f31241425f0 2024-03-29T11:20:34.220 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:34.220 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:34.220 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 10 monclient(hunting): dump: 2024-03-29T11:20:34.220 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:34.220 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:34.220 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.220 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.220 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:34.220 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:34.220 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:34.220 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:34.221 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:34.221 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:34.221 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 1 -- 172.21.15.196:0/3571397586 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2005842078 0 0) 0x7f3104004eb0 con 0x7f31241ccb70 2024-03-29T11:20:34.221 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 1 -- 172.21.15.196:0/3571397586 >> v1:172.21.15.196:6789/0 conn(0x7f312409ac40 legacy=0x7f31241da690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.221 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 1 -- 172.21.15.196:0/3571397586 >> v1:172.21.15.86:6789/0 conn(0x7f31241425f0 legacy=0x7f31241e29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.221 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:34.221 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 1 monclient: found mon.c 2024-03-29T11:20:34.221 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:34.221 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:20:34.221 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 1 -- 172.21.15.196:0/3571397586 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31240b1af0 con 0x7f31241ccb70 2024-03-29T11:20:34.221 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:34.221 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:34.221 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:34.221 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3128e25640 5 monclient: authenticate success, global_id 4109 2024-03-29T11:20:34.222 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3128e25640 10 monclient: _renew_subs 2024-03-29T11:20:34.222 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3128e25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:20:34.222 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3128e25640 1 -- 172.21.15.196:0/3571397586 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31241cc300 con 0x7f31241ccb70 2024-03-29T11:20:34.222 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3128e25640 10 client.4109.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:34.222 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3128e25640 10 monclient: _renew_subs 2024-03-29T11:20:34.222 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3128e25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:20:34.222 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3128e25640 1 -- 172.21.15.196:0/3571397586 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f31241daf80 con 0x7f31241ccb70 2024-03-29T11:20:34.222 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3128e25640 1 librados: init done 2024-03-29T11:20:34.222 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 1 -- 172.21.15.196:0/3571397586 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3118001cb0 con 0x7f31241ccb70 2024-03-29T11:20:34.222 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:34.222 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 1 -- 172.21.15.196:0/3571397586 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f3118003790 con 0x7f31241ccb70 2024-03-29T11:20:34.222 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:34.223 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3120d99640 4 set_mon_vals no callback set 2024-03-29T11:20:34.223 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:34.223 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 10 monclient: dump: 2024-03-29T11:20:34.223 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:34.223 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:34.223 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.223 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.223 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:34.223 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:34.223 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:34.223 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:34.223 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:34.223 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:34.224 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 1 -- 172.21.15.196:0/3571397586 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f3118003c80 con 0x7f31241ccb70 2024-03-29T11:20:34.224 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 1 -- 172.21.15.196:0/3571397586 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(2..2 src has 1..2) v4 ==== 725+0+0 (unknown 3205701424 0 0) 0x7f31180052a0 con 0x7f31241ccb70 2024-03-29T11:20:34.224 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 10 client.4109.objecter ms_dispatch 0x7f3124000b90 osd_map(2..2 src has 1..2) v4 2024-03-29T11:20:34.224 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 3 client.4109.objecter handle_osd_map got epochs [2,2] > 0 2024-03-29T11:20:34.224 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 3 client.4109.objecter handle_osd_map decoding full epoch 2 2024-03-29T11:20:34.224 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.222+0000 7f3112ffd640 20 client.4109.objecter dump_active .. 0 homeless 2024-03-29T11:20:34.224 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.223+0000 7f3128e25640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:34.224 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.223+0000 7f3128e25640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:34.224 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.223+0000 7f3128e25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:20:34.224 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.223+0000 7f3128e25640 1 -- 172.21.15.196:0/3571397586 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3108005080 con 0x7f31241ccb70 2024-03-29T11:20:34.224 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.225+0000 7f3112ffd640 1 -- 172.21.15.196:0/3571397586 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f3118004220 con 0x7f31241ccb70 2024-03-29T11:20:34.224 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.225+0000 7f3112ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:34.224 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.225+0000 7f3112ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:34.274 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.297+0000 7f3128e25640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "69e25ec4-1959-46d9-8d55-eb4e52eaf6f1", "id": 0}] 2024-03-29T11:20:34.274 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.297+0000 7f3128e25640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "69e25ec4-1959-46d9-8d55-eb4e52eaf6f1", "id": 0}] 2024-03-29T11:20:34.274 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.297+0000 7f3128e25640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:20:34.274 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.297+0000 7f3128e25640 1 -- 172.21.15.196:0/3571397586 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd new", "uuid": "69e25ec4-1959-46d9-8d55-eb4e52eaf6f1", "id": 0} v 0) v1 -- 0x7f31080058b0 con 0x7f31241ccb70 2024-03-29T11:20:34.277 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.300+0000 7f3112ffd640 1 -- 172.21.15.196:0/3571397586 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "69e25ec4-1959-46d9-8d55-eb4e52eaf6f1", "id": 0}]=0 v3) v1 ==== 112+0+2 (unknown 4261318141 0 732707515) 0x7f3118004490 con 0x7f31241ccb70 2024-03-29T11:20:34.277 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.300+0000 7f3112ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "69e25ec4-1959-46d9-8d55-eb4e52eaf6f1", "id": 0}] 2024-03-29T11:20:34.277 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.300+0000 7f3112ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:34.277 INFO:teuthology.orchestra.run.smithi196.stdout:0 2024-03-29T11:20:34.279 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.301+0000 7f3128e25640 10 librados: watch_flush enter 2024-03-29T11:20:34.279 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.301+0000 7f3128e25640 10 librados: watch_flush exit 2024-03-29T11:20:34.279 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.301+0000 7f3128e25640 20 client.4109.objecter shutdown clearing up homeless session... 2024-03-29T11:20:34.279 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.301+0000 7f3128e25640 10 client.4109.objecter successfully canceled tick 2024-03-29T11:20:34.279 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.301+0000 7f3128e25640 10 monclient: shutdown 2024-03-29T11:20:34.279 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.301+0000 7f3128e25640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:34.279 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.301+0000 7f3128e25640 1 -- 172.21.15.196:0/3571397586 >> v1:172.21.15.86:6790/0 conn(0x7f31241ccb70 legacy=0x7f312413adc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.279 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.301+0000 7f3128e25640 1 -- 172.21.15.196:0/3571397586 shutdown_connections 2024-03-29T11:20:34.279 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.301+0000 7f3128e25640 1 -- 172.21.15.196:0/3571397586 >> 172.21.15.196:0/3571397586 conn(0x7f3124066b00 msgr2=0x7f3124066ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:34.279 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.301+0000 7f3128e25640 1 -- 172.21.15.196:0/3571397586 shutdown_connections 2024-03-29T11:20:34.280 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.301+0000 7f3128e25640 1 -- 172.21.15.196:0/3571397586 wait complete. 2024-03-29T11:20:34.280 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.301+0000 7f3128e25640 1 librados: shutdown 2024-03-29T11:20:34.288 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 8378e0ad-8983-46ba-a2f1-24b7fc3c7df4 1 2024-03-29T11:20:34.387 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.410+0000 7f4219d5d640 10 monclient: get_monmap_and_config 2024-03-29T11:20:34.387 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.410+0000 7f4219d5d640 10 monclient: build_initial_monmap 2024-03-29T11:20:34.387 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.410+0000 7f4219d5d640 10 monclient: monmap: 2024-03-29T11:20:34.388 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-29T11:20:34.388 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:34.388 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:20:34.412003+0000 2024-03-29T11:20:34.388 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:20:34.412003+0000 2024-03-29T11:20:34.388 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:34.388 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:34.388 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:20:34.388 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:20:34.388 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:20:34.388 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:34.396 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.411+0000 7f4219d5d640 1 Processor -- start 2024-03-29T11:20:34.396 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.411+0000 7f4219d5d640 1 -- start start 2024-03-29T11:20:34.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.411+0000 7f4219d5d640 10 monclient: init 2024-03-29T11:20:34.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.412+0000 7f4219d5d640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:34.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.412+0000 7f4219d5d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:34.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.412+0000 7f4219d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f42140b0ad0 addr v1:172.21.15.86:6790/0 2024-03-29T11:20:34.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.412+0000 7f4219d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f4214142320 addr v1:172.21.15.86:6789/0 2024-03-29T11:20:34.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.412+0000 7f4219d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f421406f430 addr v1:172.21.15.196:6789/0 2024-03-29T11:20:34.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.412+0000 7f4219d5d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4214142780 con 0x7f4214142320 2024-03-29T11:20:34.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.412+0000 7f4219d5d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4214072690 con 0x7f421406f430 2024-03-29T11:20:34.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.412+0000 7f4219d5d640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4214071230 con 0x7f42140b0ad0 2024-03-29T11:20:34.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.412+0000 7f4219d5d640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:34.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.412+0000 7f4219d5d640 10 monclient(hunting): authenticate will time out at 879.700610s 2024-03-29T11:20:34.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.412+0000 7f4213fff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f421406f430 0x7f421414a6c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:51488/0 (socket says 172.21.15.196:51488) 2024-03-29T11:20:34.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.412+0000 7f4213fff640 1 -- 172.21.15.196:0/1395481482 learned_addr learned my addr 172.21.15.196:0/1395481482 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-29T11:20:34.397 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.412+0000 7f4212ffd640 1 -- 172.21.15.196:0/1395481482 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2215092864 0 0) 0x7f4214072690 con 0x7f421406f430 2024-03-29T11:20:34.398 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.412+0000 7f4212ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:34.398 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.412+0000 7f4212ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:34.398 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.412+0000 7f4212ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:34.398 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.412+0000 7f4212ffd640 10 monclient(hunting): my global_id is 4117 2024-03-29T11:20:34.398 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.412+0000 7f4212ffd640 1 -- 172.21.15.196:0/1395481482 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4200003520 con 0x7f421406f430 2024-03-29T11:20:34.398 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4212ffd640 1 -- 172.21.15.196:0/1395481482 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 978771542 0 0) 0x7f4214071230 con 0x7f42140b0ad0 2024-03-29T11:20:34.398 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4212ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:34.398 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4212ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:34.398 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4212ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:34.398 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4212ffd640 10 monclient(hunting): my global_id is 4112 2024-03-29T11:20:34.398 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4212ffd640 1 -- 172.21.15.196:0/1395481482 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4214072690 con 0x7f42140b0ad0 2024-03-29T11:20:34.398 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4212ffd640 1 -- 172.21.15.196:0/1395481482 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2968250496 0 0) 0x7f4214142780 con 0x7f4214142320 2024-03-29T11:20:34.398 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4212ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:34.398 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4212ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:34.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4212ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:34.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4212ffd640 10 monclient(hunting): my global_id is 4119 2024-03-29T11:20:34.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4212ffd640 1 -- 172.21.15.196:0/1395481482 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4214071230 con 0x7f4214142320 2024-03-29T11:20:34.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4212ffd640 1 -- 172.21.15.196:0/1395481482 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 508908195 0 0) 0x7f4200003520 con 0x7f421406f430 2024-03-29T11:20:34.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4212ffd640 1 -- 172.21.15.196:0/1395481482 >> v1:172.21.15.86:6790/0 conn(0x7f42140b0ad0 legacy=0x7f4214071f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4212ffd640 1 -- 172.21.15.196:0/1395481482 >> v1:172.21.15.86:6789/0 conn(0x7f4214142320 legacy=0x7f421406ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4212ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:34.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4212ffd640 1 monclient: found mon.noname-c 2024-03-29T11:20:34.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4212ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:34.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4212ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T11:20:34.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4212ffd640 1 -- 172.21.15.196:0/1395481482 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42140714a0 con 0x7f421406f430 2024-03-29T11:20:34.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4212ffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:34.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4212ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:34.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4212ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:34.399 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4219d5d640 5 monclient: authenticate success, global_id 4117 2024-03-29T11:20:34.400 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4212ffd640 1 -- 172.21.15.196:0/1395481482 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f420c001a10 con 0x7f421406f430 2024-03-29T11:20:34.400 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4219d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:34.400 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4212ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:34.400 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4212ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:34.400 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4212ffd640 10 monclient: dump: 2024-03-29T11:20:34.400 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:34.400 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:34.400 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.400 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.400 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:34.400 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:34.400 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:34.400 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:34.400 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:34.400 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:34.400 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4219d5d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:34.401 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4212ffd640 1 -- 172.21.15.196:0/1395481482 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f420c001bf0 con 0x7f421406f430 2024-03-29T11:20:34.401 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4212ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:34.401 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4219d5d640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:34.401 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4219d5d640 4 set_mon_vals no callback set 2024-03-29T11:20:34.401 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4212ffd640 1 -- 172.21.15.196:0/1395481482 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f420c002000 con 0x7f421406f430 2024-03-29T11:20:34.401 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4212ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:34.401 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4212ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:34.401 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4212ffd640 10 monclient: dump: 2024-03-29T11:20:34.401 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:34.401 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:34.401 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.401 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.401 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:34.401 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:34.401 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:34.402 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:34.402 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:34.402 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:34.402 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4219d5d640 10 monclient: shutdown 2024-03-29T11:20:34.402 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4219d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:34.402 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4219d5d640 1 -- 172.21.15.196:0/1395481482 >> v1:172.21.15.196:6789/0 conn(0x7f421406f430 legacy=0x7f421414a6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.402 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4219d5d640 1 -- 172.21.15.196:0/1395481482 shutdown_connections 2024-03-29T11:20:34.402 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4219d5d640 1 -- 172.21.15.196:0/1395481482 >> 172.21.15.196:0/1395481482 conn(0x7f42140b4970 msgr2=0x7f42140b4d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:34.402 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4219d5d640 1 -- 172.21.15.196:0/1395481482 shutdown_connections 2024-03-29T11:20:34.402 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.413+0000 7f4219d5d640 1 -- 172.21.15.196:0/1395481482 wait complete. 2024-03-29T11:20:34.402 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.414+0000 7f4219d5d640 10 monclient: build_initial_monmap 2024-03-29T11:20:34.402 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.414+0000 7f4219d5d640 10 monclient: monmap: 2024-03-29T11:20:34.402 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-29T11:20:34.402 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:34.403 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 0.000000 2024-03-29T11:20:34.403 INFO:teuthology.orchestra.run.smithi196.stderr:created 0.000000 2024-03-29T11:20:34.403 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:34.403 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:34.403 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:20:34.403 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:20:34.403 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:20:34.403 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:34.403 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.414+0000 7f4219d5d640 1 librados: starting msgr at 2024-03-29T11:20:34.403 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.414+0000 7f4219d5d640 1 librados: starting objecter 2024-03-29T11:20:34.403 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.414+0000 7f4219d5d640 1 Processor -- start 2024-03-29T11:20:34.403 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.414+0000 7f4219d5d640 1 -- start start 2024-03-29T11:20:34.403 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.414+0000 7f4219d5d640 1 librados: setting wanted keys 2024-03-29T11:20:34.403 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.414+0000 7f4219d5d640 1 librados: calling monclient init 2024-03-29T11:20:34.403 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.414+0000 7f4219d5d640 10 monclient: init 2024-03-29T11:20:34.403 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f4219d5d640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:34.404 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f4219d5d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:34.404 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f4219d5d640 10 monclient(hunting): picked mon.noname-a con 0x7f421406f430 addr v1:172.21.15.86:6789/0 2024-03-29T11:20:34.404 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f4219d5d640 10 monclient(hunting): picked mon.noname-b con 0x7f42140b0ad0 addr v1:172.21.15.196:6789/0 2024-03-29T11:20:34.404 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f4219d5d640 10 monclient(hunting): picked mon.noname-c con 0x7f4214142320 addr v1:172.21.15.86:6790/0 2024-03-29T11:20:34.404 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f4219d5d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42141d7bc0 con 0x7f421406f430 2024-03-29T11:20:34.404 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f4219d5d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42140aadb0 con 0x7f42140b0ad0 2024-03-29T11:20:34.404 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f4219d5d640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42141d80c0 con 0x7f4214142320 2024-03-29T11:20:34.404 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f4219d5d640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:34.404 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f4219d5d640 10 monclient(hunting): authenticate will time out at 879.703541s 2024-03-29T11:20:34.404 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f420bfff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f42140b0ad0 0x7f4214134c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:51492/0 (socket says 172.21.15.196:51492) 2024-03-29T11:20:34.404 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f42137fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f421406f430 0x7f42140aa6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.196:39278/0 (socket says 172.21.15.196:39278) 2024-03-29T11:20:34.404 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f420bfff640 1 -- 172.21.15.196:0/2253661880 learned_addr learned my addr 172.21.15.196:0/2253661880 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-29T11:20:34.404 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f4210ff9640 10 client.?.objecter ms_handle_connect 0x7f42140b0ad0 2024-03-29T11:20:34.404 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f4210ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:34.404 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f4210ff9640 10 client.?.objecter ms_handle_connect 0x7f421406f430 2024-03-29T11:20:34.405 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f4210ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:34.405 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f4210ff9640 10 client.?.objecter ms_handle_connect 0x7f4214142320 2024-03-29T11:20:34.405 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f4210ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:34.405 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f4210ff9640 1 -- 172.21.15.196:0/2253661880 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 941408724 0 0) 0x7f42140aadb0 con 0x7f42140b0ad0 2024-03-29T11:20:34.405 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f4210ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:34.405 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f4210ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:34.405 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f4210ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:34.405 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f4210ff9640 10 monclient(hunting): my global_id is 4120 2024-03-29T11:20:34.405 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f4210ff9640 1 -- 172.21.15.196:0/2253661880 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f41f4004eb0 con 0x7f42140b0ad0 2024-03-29T11:20:34.405 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f4210ff9640 1 -- 172.21.15.196:0/2253661880 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2253821595 0 0) 0x7f42141d7bc0 con 0x7f421406f430 2024-03-29T11:20:34.405 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f4210ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:34.405 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f4210ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:34.405 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f4210ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:34.405 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f4210ff9640 10 monclient(hunting): my global_id is 4122 2024-03-29T11:20:34.405 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f4210ff9640 1 -- 172.21.15.196:0/2253661880 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42140aadb0 con 0x7f421406f430 2024-03-29T11:20:34.406 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f4210ff9640 1 -- 172.21.15.196:0/2253661880 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3228155267 0 0) 0x7f42141d80c0 con 0x7f4214142320 2024-03-29T11:20:34.406 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f4210ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:34.406 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f4210ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:34.406 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f4210ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:34.406 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f4210ff9640 10 monclient(hunting): my global_id is 4115 2024-03-29T11:20:34.406 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.415+0000 7f4210ff9640 1 -- 172.21.15.196:0/2253661880 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f42141d7bc0 con 0x7f4214142320 2024-03-29T11:20:34.406 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 1 -- 172.21.15.196:0/2253661880 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2206919795 0 0) 0x7f41f4004eb0 con 0x7f42140b0ad0 2024-03-29T11:20:34.406 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 1 -- 172.21.15.196:0/2253661880 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42141d80c0 con 0x7f42140b0ad0 2024-03-29T11:20:34.406 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 1 -- 172.21.15.196:0/2253661880 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f41fc001910 con 0x7f42140b0ad0 2024-03-29T11:20:34.406 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:34.406 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:34.406 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 10 monclient(hunting): dump: 2024-03-29T11:20:34.406 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:34.406 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:34.406 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.407 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.407 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:34.407 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:34.407 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:34.407 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:34.407 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:34.407 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:34.407 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:34.407 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 1 -- 172.21.15.196:0/2253661880 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 14990925 0 0) 0x7f42141d7bc0 con 0x7f4214142320 2024-03-29T11:20:34.407 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 1 -- 172.21.15.196:0/2253661880 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f41f4004eb0 con 0x7f4214142320 2024-03-29T11:20:34.407 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 1 -- 172.21.15.196:0/2253661880 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f420c001f00 con 0x7f4214142320 2024-03-29T11:20:34.407 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:34.407 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:34.407 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 10 monclient(hunting): dump: 2024-03-29T11:20:34.407 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:34.407 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:34.408 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.408 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.408 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:34.408 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:34.408 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:34.408 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:34.408 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:34.408 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:34.408 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 1 -- 172.21.15.196:0/2253661880 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 689525443 0 0) 0x7f42140aadb0 con 0x7f421406f430 2024-03-29T11:20:34.408 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 1 -- 172.21.15.196:0/2253661880 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f42141d7bc0 con 0x7f421406f430 2024-03-29T11:20:34.408 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 1 -- 172.21.15.196:0/2253661880 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f42040018b0 con 0x7f421406f430 2024-03-29T11:20:34.408 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:34.408 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:34.408 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 10 monclient(hunting): dump: 2024-03-29T11:20:34.408 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:34.409 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:34.409 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.409 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.409 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:34.409 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:34.409 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:34.409 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:34.409 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:34.409 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:34.409 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 1 -- 172.21.15.196:0/2253661880 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 246684764 0 0) 0x7f42141d80c0 con 0x7f42140b0ad0 2024-03-29T11:20:34.409 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 1 -- 172.21.15.196:0/2253661880 >> v1:172.21.15.86:6790/0 conn(0x7f4214142320 legacy=0x7f42141353a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.409 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 1 -- 172.21.15.196:0/2253661880 >> v1:172.21.15.86:6789/0 conn(0x7f421406f430 legacy=0x7f42140aa6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.409 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:20:34.409 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 1 monclient: found mon.b 2024-03-29T11:20:34.409 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:34.410 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:20:34.410 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 1 -- 172.21.15.196:0/2253661880 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42141d8250 con 0x7f42140b0ad0 2024-03-29T11:20:34.410 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 10 monclient: _finish_auth 0 2024-03-29T11:20:34.410 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:20:34.410 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:34.410 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4219d5d640 5 monclient: authenticate success, global_id 4120 2024-03-29T11:20:34.410 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4219d5d640 10 monclient: _renew_subs 2024-03-29T11:20:34.410 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4219d5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:20:34.410 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4219d5d640 1 -- 172.21.15.196:0/2253661880 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4214135d60 con 0x7f42140b0ad0 2024-03-29T11:20:34.410 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4219d5d640 10 client.4120.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:34.410 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4219d5d640 10 monclient: _renew_subs 2024-03-29T11:20:34.410 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4219d5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:20:34.410 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4219d5d640 1 -- 172.21.15.196:0/2253661880 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4214141580 con 0x7f42140b0ad0 2024-03-29T11:20:34.410 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4219d5d640 1 librados: init done 2024-03-29T11:20:34.410 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 1 -- 172.21.15.196:0/2253661880 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f41fc003610 con 0x7f42140b0ad0 2024-03-29T11:20:34.411 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:34.411 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 1 -- 172.21.15.196:0/2253661880 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f41fc003960 con 0x7f42140b0ad0 2024-03-29T11:20:34.411 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:34.411 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:34.411 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f42127fc640 4 set_mon_vals no callback set 2024-03-29T11:20:34.411 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 10 monclient: dump: 2024-03-29T11:20:34.411 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:34.411 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:34.411 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.411 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.411 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:34.411 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:34.411 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:34.411 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:34.411 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:34.412 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:34.412 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 1 -- 172.21.15.196:0/2253661880 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f41fc003f10 con 0x7f42140b0ad0 2024-03-29T11:20:34.412 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 1 -- 172.21.15.196:0/2253661880 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(3..3 src has 1..3) v4 ==== 840+0+0 (unknown 268374434 0 0) 0x7f41fc006600 con 0x7f42140b0ad0 2024-03-29T11:20:34.412 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 10 client.4120.objecter ms_dispatch 0x7f4214000b90 osd_map(3..3 src has 1..3) v4 2024-03-29T11:20:34.412 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 3 client.4120.objecter handle_osd_map got epochs [3,3] > 0 2024-03-29T11:20:34.412 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 3 client.4120.objecter handle_osd_map decoding full epoch 3 2024-03-29T11:20:34.412 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.416+0000 7f4210ff9640 20 client.4120.objecter dump_active .. 0 homeless 2024-03-29T11:20:34.412 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.417+0000 7f4219d5d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:34.412 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.417+0000 7f4219d5d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:34.412 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.417+0000 7f4219d5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:20:34.412 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.417+0000 7f4219d5d640 1 -- 172.21.15.196:0/2253661880 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41f8005080 con 0x7f42140b0ad0 2024-03-29T11:20:34.412 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.419+0000 7f4210ff9640 1 -- 172.21.15.196:0/2253661880 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f41fc0043f0 con 0x7f42140b0ad0 2024-03-29T11:20:34.412 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.419+0000 7f4210ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:34.412 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.419+0000 7f4210ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:34.485 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.509+0000 7f4219d5d640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "8378e0ad-8983-46ba-a2f1-24b7fc3c7df4", "id": 1}] 2024-03-29T11:20:34.486 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.509+0000 7f4219d5d640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "8378e0ad-8983-46ba-a2f1-24b7fc3c7df4", "id": 1}] 2024-03-29T11:20:34.486 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.509+0000 7f4219d5d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:20:34.486 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.509+0000 7f4219d5d640 1 -- 172.21.15.196:0/2253661880 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd new", "uuid": "8378e0ad-8983-46ba-a2f1-24b7fc3c7df4", "id": 1} v 0) v1 -- 0x7f41f80058b0 con 0x7f42140b0ad0 2024-03-29T11:20:34.493 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.516+0000 7f4210ff9640 1 -- 172.21.15.196:0/2253661880 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "8378e0ad-8983-46ba-a2f1-24b7fc3c7df4", "id": 1}]=0 v4) v1 ==== 112+0+2 (unknown 2796883110 0 940483276) 0x7f41fc005890 con 0x7f42140b0ad0 2024-03-29T11:20:34.493 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.516+0000 7f4210ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "8378e0ad-8983-46ba-a2f1-24b7fc3c7df4", "id": 1}] 2024-03-29T11:20:34.493 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.516+0000 7f4210ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:34.494 INFO:teuthology.orchestra.run.smithi196.stdout:1 2024-03-29T11:20:34.495 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.517+0000 7f4219d5d640 10 librados: watch_flush enter 2024-03-29T11:20:34.496 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.517+0000 7f4219d5d640 10 librados: watch_flush exit 2024-03-29T11:20:34.496 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.517+0000 7f4219d5d640 20 client.4120.objecter shutdown clearing up homeless session... 2024-03-29T11:20:34.496 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.517+0000 7f4219d5d640 10 client.4120.objecter successfully canceled tick 2024-03-29T11:20:34.496 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.517+0000 7f4219d5d640 10 monclient: shutdown 2024-03-29T11:20:34.496 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.517+0000 7f4219d5d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:34.496 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.517+0000 7f4219d5d640 1 -- 172.21.15.196:0/2253661880 >> v1:172.21.15.196:6789/0 conn(0x7f42140b0ad0 legacy=0x7f4214134c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.496 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.517+0000 7f4219d5d640 1 -- 172.21.15.196:0/2253661880 shutdown_connections 2024-03-29T11:20:34.496 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.517+0000 7f4219d5d640 1 -- 172.21.15.196:0/2253661880 >> 172.21.15.196:0/2253661880 conn(0x7f42140b4970 msgr2=0x7f421406ea90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:34.496 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.517+0000 7f4219d5d640 1 -- 172.21.15.196:0/2253661880 shutdown_connections 2024-03-29T11:20:34.496 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.517+0000 7f4219d5d640 1 -- 172.21.15.196:0/2253661880 wait complete. 2024-03-29T11:20:34.496 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.517+0000 7f4219d5d640 1 librados: shutdown 2024-03-29T11:20:34.505 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new a5d7c244-1f0f-4632-8246-1226c2c180d5 2 2024-03-29T11:20:34.583 INFO:tasks.ceph.mgr.y.smithi086.stderr:2024-03-29T11:20:34.606+0000 7f4bd1e42040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:20:34.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.628+0000 7feda8b46640 10 monclient: get_monmap_and_config 2024-03-29T11:20:34.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.628+0000 7feda8b46640 10 monclient: build_initial_monmap 2024-03-29T11:20:34.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.628+0000 7feda8b46640 10 monclient: monmap: 2024-03-29T11:20:34.605 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-29T11:20:34.605 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:34.605 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:20:34.629132+0000 2024-03-29T11:20:34.605 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:20:34.629132+0000 2024-03-29T11:20:34.605 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:34.605 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:34.605 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:20:34.605 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:20:34.605 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:20:34.605 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:34.612 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.628+0000 7feda8b46640 1 Processor -- start 2024-03-29T11:20:34.612 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.628+0000 7feda8b46640 1 -- start start 2024-03-29T11:20:34.612 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.628+0000 7feda8b46640 10 monclient: init 2024-03-29T11:20:34.612 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.629+0000 7feda8b46640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:34.612 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.629+0000 7feda8b46640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:34.612 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.629+0000 7feda8b46640 10 monclient(hunting): picked mon.noname-a con 0x7feda40b8240 addr v1:172.21.15.86:6789/0 2024-03-29T11:20:34.612 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.629+0000 7feda8b46640 10 monclient(hunting): picked mon.noname-c con 0x7feda40b1f60 addr v1:172.21.15.196:6789/0 2024-03-29T11:20:34.613 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.629+0000 7feda8b46640 10 monclient(hunting): picked mon.noname-b con 0x7feda409c8c0 addr v1:172.21.15.86:6790/0 2024-03-29T11:20:34.613 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.629+0000 7feda8b46640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda409bd80 con 0x7feda40b8240 2024-03-29T11:20:34.613 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.629+0000 7feda8b46640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda409b960 con 0x7feda40b1f60 2024-03-29T11:20:34.613 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.629+0000 7feda8b46640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda40b2ec0 con 0x7feda409c8c0 2024-03-29T11:20:34.613 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.629+0000 7feda8b46640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:34.613 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.629+0000 7feda8b46640 10 monclient(hunting): authenticate will time out at 879.917535s 2024-03-29T11:20:34.613 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.629+0000 7feda1d9b640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7feda40b1f60 0x7feda409c1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:51498/0 (socket says 172.21.15.196:51498) 2024-03-29T11:20:34.613 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.629+0000 7feda1d9b640 1 -- 172.21.15.196:0/1196374251 learned_addr learned my addr 172.21.15.196:0/1196374251 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-29T11:20:34.613 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.629+0000 7feda159a640 1 -- 172.21.15.196:0/1196374251 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2141186818 0 0) 0x7feda409b960 con 0x7feda40b1f60 2024-03-29T11:20:34.613 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.629+0000 7feda159a640 10 monclient(hunting): _negotiate 2024-03-29T11:20:34.613 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.629+0000 7feda159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:34.613 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.629+0000 7feda159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:34.613 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.629+0000 7feda159a640 10 monclient(hunting): my global_id is 4123 2024-03-29T11:20:34.614 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.629+0000 7feda159a640 1 -- 172.21.15.196:0/1196374251 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed8c003520 con 0x7feda40b1f60 2024-03-29T11:20:34.614 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.629+0000 7feda159a640 1 -- 172.21.15.196:0/1196374251 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1176751394 0 0) 0x7feda40b2ec0 con 0x7feda409c8c0 2024-03-29T11:20:34.614 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.629+0000 7feda159a640 10 monclient(hunting): _negotiate 2024-03-29T11:20:34.614 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.629+0000 7feda159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:34.614 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.629+0000 7feda159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:34.614 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.629+0000 7feda159a640 10 monclient(hunting): my global_id is 4118 2024-03-29T11:20:34.614 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.629+0000 7feda159a640 1 -- 172.21.15.196:0/1196374251 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda409b960 con 0x7feda409c8c0 2024-03-29T11:20:34.614 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.629+0000 7feda159a640 1 -- 172.21.15.196:0/1196374251 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2109567465 0 0) 0x7feda409bd80 con 0x7feda40b8240 2024-03-29T11:20:34.614 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.629+0000 7feda159a640 10 monclient(hunting): _negotiate 2024-03-29T11:20:34.614 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.629+0000 7feda159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:34.614 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.629+0000 7feda159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:34.615 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.629+0000 7feda159a640 10 monclient(hunting): my global_id is 4125 2024-03-29T11:20:34.615 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.629+0000 7feda159a640 1 -- 172.21.15.196:0/1196374251 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda40b2ec0 con 0x7feda40b8240 2024-03-29T11:20:34.615 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.630+0000 7feda159a640 1 -- 172.21.15.196:0/1196374251 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 590981926 0 0) 0x7fed8c003520 con 0x7feda40b1f60 2024-03-29T11:20:34.615 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.630+0000 7feda159a640 1 -- 172.21.15.196:0/1196374251 >> v1:172.21.15.86:6790/0 conn(0x7feda409c8c0 legacy=0x7feda414d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.615 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.630+0000 7feda159a640 1 -- 172.21.15.196:0/1196374251 >> v1:172.21.15.86:6789/0 conn(0x7feda40b8240 legacy=0x7feda413f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.615 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.630+0000 7feda159a640 10 monclient: _finish_hunting 0 2024-03-29T11:20:34.615 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.630+0000 7feda159a640 1 monclient: found mon.noname-c 2024-03-29T11:20:34.615 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.630+0000 7feda159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:34.615 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.630+0000 7feda159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T11:20:34.615 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.630+0000 7feda159a640 1 -- 172.21.15.196:0/1196374251 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda413fb60 con 0x7feda40b1f60 2024-03-29T11:20:34.615 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.630+0000 7feda159a640 10 monclient: _finish_auth 0 2024-03-29T11:20:34.615 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.630+0000 7feda159a640 10 monclient: _check_auth_tickets 2024-03-29T11:20:34.615 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.630+0000 7feda159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:34.616 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.630+0000 7feda159a640 1 -- 172.21.15.196:0/1196374251 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fed98001690 con 0x7feda40b1f60 2024-03-29T11:20:34.616 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.630+0000 7feda8b46640 5 monclient: authenticate success, global_id 4123 2024-03-29T11:20:34.616 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.630+0000 7feda8b46640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:34.616 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.630+0000 7feda159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:34.616 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.630+0000 7feda159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:34.616 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.630+0000 7feda159a640 10 monclient: dump: 2024-03-29T11:20:34.616 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:34.616 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:34.616 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.616 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.616 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:34.616 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:34.617 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:34.617 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:34.617 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:34.617 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:34.617 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.630+0000 7feda8b46640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:34.617 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.630+0000 7feda159a640 1 -- 172.21.15.196:0/1196374251 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed98001870 con 0x7feda40b1f60 2024-03-29T11:20:34.617 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.630+0000 7feda159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:34.617 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.630+0000 7feda159a640 1 -- 172.21.15.196:0/1196374251 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fed98001cc0 con 0x7feda40b1f60 2024-03-29T11:20:34.617 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.630+0000 7feda8b46640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:34.617 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.630+0000 7feda8b46640 4 set_mon_vals no callback set 2024-03-29T11:20:34.617 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.630+0000 7feda159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:34.617 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.630+0000 7feda159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:34.618 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.630+0000 7feda159a640 10 monclient: dump: 2024-03-29T11:20:34.618 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:34.618 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:34.618 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.618 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.618 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:34.618 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:34.618 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:34.618 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:34.618 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:34.618 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:34.618 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.630+0000 7feda8b46640 10 monclient: shutdown 2024-03-29T11:20:34.618 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.630+0000 7feda8b46640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:34.619 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.630+0000 7feda8b46640 1 -- 172.21.15.196:0/1196374251 >> v1:172.21.15.196:6789/0 conn(0x7feda40b1f60 legacy=0x7feda409c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.619 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.630+0000 7feda8b46640 1 -- 172.21.15.196:0/1196374251 shutdown_connections 2024-03-29T11:20:34.619 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.630+0000 7feda8b46640 1 -- 172.21.15.196:0/1196374251 >> 172.21.15.196:0/1196374251 conn(0x7feda4066b00 msgr2=0x7feda4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:34.619 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.631+0000 7feda8b46640 1 -- 172.21.15.196:0/1196374251 shutdown_connections 2024-03-29T11:20:34.619 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.631+0000 7feda8b46640 1 -- 172.21.15.196:0/1196374251 wait complete. 2024-03-29T11:20:34.619 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.631+0000 7feda8b46640 10 monclient: build_initial_monmap 2024-03-29T11:20:34.619 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.631+0000 7feda8b46640 10 monclient: monmap: 2024-03-29T11:20:34.619 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-29T11:20:34.619 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:34.619 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 0.000000 2024-03-29T11:20:34.619 INFO:teuthology.orchestra.run.smithi196.stderr:created 0.000000 2024-03-29T11:20:34.619 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:34.620 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:34.620 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:20:34.620 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:20:34.620 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:20:34.620 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:34.620 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.631+0000 7feda8b46640 1 librados: starting msgr at 2024-03-29T11:20:34.620 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.631+0000 7feda8b46640 1 librados: starting objecter 2024-03-29T11:20:34.620 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.632+0000 7feda8b46640 1 Processor -- start 2024-03-29T11:20:34.620 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.632+0000 7feda8b46640 1 -- start start 2024-03-29T11:20:34.620 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.632+0000 7feda8b46640 1 librados: setting wanted keys 2024-03-29T11:20:34.620 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.632+0000 7feda8b46640 1 librados: calling monclient init 2024-03-29T11:20:34.620 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.632+0000 7feda8b46640 10 monclient: init 2024-03-29T11:20:34.620 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.632+0000 7feda8b46640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:34.621 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.632+0000 7feda8b46640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:34.621 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.632+0000 7feda8b46640 10 monclient(hunting): picked mon.noname-a con 0x7feda409c8c0 addr v1:172.21.15.86:6789/0 2024-03-29T11:20:34.621 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.632+0000 7feda8b46640 10 monclient(hunting): picked mon.noname-b con 0x7feda40b8240 addr v1:172.21.15.196:6789/0 2024-03-29T11:20:34.621 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.632+0000 7feda8b46640 10 monclient(hunting): picked mon.noname-c con 0x7feda413bf40 addr v1:172.21.15.86:6790/0 2024-03-29T11:20:34.621 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.632+0000 7feda8b46640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda4146420 con 0x7feda409c8c0 2024-03-29T11:20:34.621 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.632+0000 7feda8b46640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda4133010 con 0x7feda40b8240 2024-03-29T11:20:34.621 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.632+0000 7feda8b46640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feda4144480 con 0x7feda413bf40 2024-03-29T11:20:34.621 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.632+0000 7feda8b46640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:34.621 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.632+0000 7feda8b46640 10 monclient(hunting): authenticate will time out at 879.920695s 2024-03-29T11:20:34.621 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.632+0000 7feda1d9b640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7feda40b8240 0x7feda4142900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:51502/0 (socket says 172.21.15.196:51502) 2024-03-29T11:20:34.621 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.632+0000 7feda1d9b640 1 -- 172.21.15.196:0/2321511452 learned_addr learned my addr 172.21.15.196:0/2321511452 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-29T11:20:34.622 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T11:20:34.632+0000 7fee3a432040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-29T11:20:34.626 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.632+0000 7fed92ffd640 10 client.?.objecter ms_handle_connect 0x7feda40b8240 2024-03-29T11:20:34.626 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.632+0000 7fed92ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:34.626 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.632+0000 7fed92ffd640 10 client.?.objecter ms_handle_connect 0x7feda409c8c0 2024-03-29T11:20:34.626 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.632+0000 7fed92ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:34.626 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.632+0000 7fed92ffd640 10 client.?.objecter ms_handle_connect 0x7feda413bf40 2024-03-29T11:20:34.626 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.632+0000 7fed92ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:34.626 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.632+0000 7fed92ffd640 1 -- 172.21.15.196:0/2321511452 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4245634909 0 0) 0x7feda4133010 con 0x7feda40b8240 2024-03-29T11:20:34.626 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.632+0000 7fed92ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:34.626 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.632+0000 7fed92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:34.627 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.632+0000 7fed92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:34.627 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.632+0000 7fed92ffd640 10 monclient(hunting): my global_id is 4126 2024-03-29T11:20:34.627 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.632+0000 7fed92ffd640 1 -- 172.21.15.196:0/2321511452 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed84004eb0 con 0x7feda40b8240 2024-03-29T11:20:34.627 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 1 -- 172.21.15.196:0/2321511452 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 673235858 0 0) 0x7feda4146420 con 0x7feda409c8c0 2024-03-29T11:20:34.627 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:34.627 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:34.627 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:34.627 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 10 monclient(hunting): my global_id is 4128 2024-03-29T11:20:34.627 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 1 -- 172.21.15.196:0/2321511452 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda4133010 con 0x7feda409c8c0 2024-03-29T11:20:34.627 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 1 -- 172.21.15.196:0/2321511452 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2559754763 0 0) 0x7feda4144480 con 0x7feda413bf40 2024-03-29T11:20:34.627 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:34.627 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:34.628 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:34.628 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 10 monclient(hunting): my global_id is 4121 2024-03-29T11:20:34.628 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 1 -- 172.21.15.196:0/2321511452 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feda4146420 con 0x7feda413bf40 2024-03-29T11:20:34.628 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 1 -- 172.21.15.196:0/2321511452 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3359515676 0 0) 0x7feda4133010 con 0x7feda409c8c0 2024-03-29T11:20:34.628 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 1 -- 172.21.15.196:0/2321511452 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feda4144480 con 0x7feda409c8c0 2024-03-29T11:20:34.628 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 1 -- 172.21.15.196:0/2321511452 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fed940017d0 con 0x7feda409c8c0 2024-03-29T11:20:34.628 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:34.628 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:34.628 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 10 monclient(hunting): dump: 2024-03-29T11:20:34.628 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:34.628 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:34.628 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.629 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.629 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:34.629 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:34.629 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:34.629 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:34.629 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:34.629 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:34.629 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:34.629 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 1 -- 172.21.15.196:0/2321511452 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3606997936 0 0) 0x7fed84004eb0 con 0x7feda40b8240 2024-03-29T11:20:34.629 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 1 -- 172.21.15.196:0/2321511452 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feda4133010 con 0x7feda40b8240 2024-03-29T11:20:34.629 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 1 -- 172.21.15.196:0/2321511452 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1676603937 0 0) 0x7feda4146420 con 0x7feda413bf40 2024-03-29T11:20:34.629 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 1 -- 172.21.15.196:0/2321511452 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fed84004eb0 con 0x7feda413bf40 2024-03-29T11:20:34.629 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 1 -- 172.21.15.196:0/2321511452 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fed98001b80 con 0x7feda40b8240 2024-03-29T11:20:34.630 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:34.630 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:34.630 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 10 monclient(hunting): dump: 2024-03-29T11:20:34.630 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:34.630 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:34.630 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.630 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.630 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:34.630 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:34.630 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:34.630 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:34.630 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:34.631 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:34.631 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 1 -- 172.21.15.196:0/2321511452 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fed9c001430 con 0x7feda413bf40 2024-03-29T11:20:34.631 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:34.631 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:34.631 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 10 monclient(hunting): dump: 2024-03-29T11:20:34.631 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:34.631 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:34.631 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.631 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.631 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:34.631 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:34.631 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:34.631 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:34.632 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:34.632 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:34.632 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 1 -- 172.21.15.196:0/2321511452 <== mon.0 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 26446684 0 0) 0x7feda4144480 con 0x7feda409c8c0 2024-03-29T11:20:34.632 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 1 -- 172.21.15.196:0/2321511452 >> v1:172.21.15.86:6790/0 conn(0x7feda413bf40 legacy=0x7feda41e2110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.632 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 1 -- 172.21.15.196:0/2321511452 >> v1:172.21.15.196:6789/0 conn(0x7feda40b8240 legacy=0x7feda4142900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.632 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:34.632 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 1 monclient: found mon.a 2024-03-29T11:20:34.632 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:34.632 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:20:34.632 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 1 -- 172.21.15.196:0/2321511452 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feda4144610 con 0x7feda409c8c0 2024-03-29T11:20:34.633 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:34.633 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:34.633 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:34.633 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7feda8b46640 5 monclient: authenticate success, global_id 4128 2024-03-29T11:20:34.633 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7feda8b46640 10 monclient: _renew_subs 2024-03-29T11:20:34.633 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7feda8b46640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:20:34.633 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7feda8b46640 1 -- 172.21.15.196:0/2321511452 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feda4141b40 con 0x7feda409c8c0 2024-03-29T11:20:34.633 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7feda8b46640 10 client.4128.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:34.633 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7feda8b46640 10 monclient: _renew_subs 2024-03-29T11:20:34.633 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7feda8b46640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:20:34.633 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7feda8b46640 1 -- 172.21.15.196:0/2321511452 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feda41326b0 con 0x7feda409c8c0 2024-03-29T11:20:34.634 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7feda8b46640 1 librados: init done 2024-03-29T11:20:34.634 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 1 -- 172.21.15.196:0/2321511452 <== mon.0 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fed94001d60 con 0x7feda409c8c0 2024-03-29T11:20:34.634 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:34.634 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 1 -- 172.21.15.196:0/2321511452 <== mon.0 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fed94003610 con 0x7feda409c8c0 2024-03-29T11:20:34.634 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:34.634 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:34.634 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7feda0d99640 4 set_mon_vals no callback set 2024-03-29T11:20:34.634 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 10 monclient: dump: 2024-03-29T11:20:34.634 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:34.634 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:34.634 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.634 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.635 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:34.635 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:34.635 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:34.635 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:34.635 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:34.635 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:34.635 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 1 -- 172.21.15.196:0/2321511452 <== mon.0 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fed94003b00 con 0x7feda409c8c0 2024-03-29T11:20:34.635 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 1 -- 172.21.15.196:0/2321511452 <== mon.0 v1:172.21.15.86:6789/0 8 ==== osd_map(4..4 src has 1..4) v4 ==== 955+0+0 (unknown 2391990138 0 0) 0x7fed940041d0 con 0x7feda409c8c0 2024-03-29T11:20:34.635 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 10 client.4128.objecter ms_dispatch 0x7feda4000b90 osd_map(4..4 src has 1..4) v4 2024-03-29T11:20:34.635 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 3 client.4128.objecter handle_osd_map got epochs [4,4] > 0 2024-03-29T11:20:34.635 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 3 client.4128.objecter handle_osd_map decoding full epoch 4 2024-03-29T11:20:34.635 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.633+0000 7fed92ffd640 20 client.4128.objecter dump_active .. 0 homeless 2024-03-29T11:20:34.636 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.634+0000 7feda8b46640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:34.636 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.634+0000 7feda8b46640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:34.636 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.634+0000 7feda8b46640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:20:34.636 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.634+0000 7feda8b46640 1 -- 172.21.15.196:0/2321511452 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed88005080 con 0x7feda409c8c0 2024-03-29T11:20:34.636 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.636+0000 7fed92ffd640 1 -- 172.21.15.196:0/2321511452 <== mon.0 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fed94004460 con 0x7feda409c8c0 2024-03-29T11:20:34.636 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.636+0000 7fed92ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:34.636 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.636+0000 7fed92ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:34.637 INFO:tasks.ceph.mgr.y.smithi086.stderr:2024-03-29T11:20:34.661+0000 7f4bd1e42040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:20:34.664 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T11:20:34.687+0000 7fee3a432040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-29T11:20:34.682 INFO:tasks.ceph.mgr.y.smithi086.stderr:2024-03-29T11:20:34.705+0000 7f4bd1e42040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:20:34.683 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.706+0000 7feda8b46640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a5d7c244-1f0f-4632-8246-1226c2c180d5", "id": 2}] 2024-03-29T11:20:34.683 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.706+0000 7feda8b46640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a5d7c244-1f0f-4632-8246-1226c2c180d5", "id": 2}] 2024-03-29T11:20:34.683 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.706+0000 7feda8b46640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:20:34.683 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.706+0000 7feda8b46640 1 -- 172.21.15.196:0/2321511452 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd new", "uuid": "a5d7c244-1f0f-4632-8246-1226c2c180d5", "id": 2} v 0) v1 -- 0x7fed880058b0 con 0x7feda409c8c0 2024-03-29T11:20:34.685 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.708+0000 7fed92ffd640 1 -- 172.21.15.196:0/2321511452 <== mon.0 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a5d7c244-1f0f-4632-8246-1226c2c180d5", "id": 2}]=0 v5) v1 ==== 112+0+2 (unknown 2726900110 0 216599125) 0x7fed94003db0 con 0x7feda409c8c0 2024-03-29T11:20:34.685 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.708+0000 7fed92ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a5d7c244-1f0f-4632-8246-1226c2c180d5", "id": 2}] 2024-03-29T11:20:34.685 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.708+0000 7fed92ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:34.685 INFO:teuthology.orchestra.run.smithi196.stdout:2 2024-03-29T11:20:34.686 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.709+0000 7feda8b46640 10 librados: watch_flush enter 2024-03-29T11:20:34.687 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.709+0000 7feda8b46640 10 librados: watch_flush exit 2024-03-29T11:20:34.687 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.709+0000 7feda8b46640 20 client.4128.objecter shutdown clearing up homeless session... 2024-03-29T11:20:34.687 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.709+0000 7feda8b46640 10 client.4128.objecter successfully canceled tick 2024-03-29T11:20:34.687 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.709+0000 7feda8b46640 10 monclient: shutdown 2024-03-29T11:20:34.687 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.709+0000 7feda8b46640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:34.687 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.709+0000 7feda8b46640 1 -- 172.21.15.196:0/2321511452 >> v1:172.21.15.86:6789/0 conn(0x7feda409c8c0 legacy=0x7feda41421f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.687 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.709+0000 7feda8b46640 1 -- 172.21.15.196:0/2321511452 shutdown_connections 2024-03-29T11:20:34.687 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.709+0000 7feda8b46640 1 -- 172.21.15.196:0/2321511452 >> 172.21.15.196:0/2321511452 conn(0x7feda4066b00 msgr2=0x7feda406bac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:34.687 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.709+0000 7feda8b46640 1 -- 172.21.15.196:0/2321511452 shutdown_connections 2024-03-29T11:20:34.687 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.709+0000 7feda8b46640 1 -- 172.21.15.196:0/2321511452 wait complete. 2024-03-29T11:20:34.687 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.709+0000 7feda8b46640 1 librados: shutdown 2024-03-29T11:20:34.696 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new a4113954-1342-43e6-8807-bddf6ae9d460 3 2024-03-29T11:20:34.712 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T11:20:34.735+0000 7fee3a432040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-29T11:20:34.737 INFO:tasks.ceph.mgr.y.smithi086.stderr:2024-03-29T11:20:34.761+0000 7f4bd1e42040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:20:34.769 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T11:20:34.792+0000 7fee3a432040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-29T11:20:34.779 INFO:tasks.ceph.mgr.y.smithi086.stderr:2024-03-29T11:20:34.803+0000 7f4bd1e42040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:20:34.796 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.819+0000 7fde406d6640 10 monclient: get_monmap_and_config 2024-03-29T11:20:34.796 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.819+0000 7fde406d6640 10 monclient: build_initial_monmap 2024-03-29T11:20:34.796 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.819+0000 7fde406d6640 10 monclient: monmap: 2024-03-29T11:20:34.796 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-29T11:20:34.796 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:34.796 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:20:34.820251+0000 2024-03-29T11:20:34.796 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:20:34.820251+0000 2024-03-29T11:20:34.796 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:34.796 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:34.796 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:20:34.796 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:20:34.796 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:20:34.797 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:34.797 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.819+0000 7fde406d6640 1 Processor -- start 2024-03-29T11:20:34.797 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.819+0000 7fde406d6640 1 -- start start 2024-03-29T11:20:34.797 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.819+0000 7fde406d6640 10 monclient: init 2024-03-29T11:20:34.804 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.820+0000 7fde406d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:34.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.820+0000 7fde406d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:34.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.820+0000 7fde406d6640 10 monclient(hunting): picked mon.noname-b con 0x7fde3809ac40 addr v1:172.21.15.86:6790/0 2024-03-29T11:20:34.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.820+0000 7fde406d6640 10 monclient(hunting): picked mon.noname-a con 0x7fde3809c870 addr v1:172.21.15.86:6789/0 2024-03-29T11:20:34.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.820+0000 7fde406d6640 10 monclient(hunting): picked mon.noname-c con 0x7fde380b4360 addr v1:172.21.15.196:6789/0 2024-03-29T11:20:34.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.820+0000 7fde406d6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde3809dec0 con 0x7fde3809c870 2024-03-29T11:20:34.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.820+0000 7fde406d6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde3809da30 con 0x7fde380b4360 2024-03-29T11:20:34.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.820+0000 7fde406d6640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde380b39c0 con 0x7fde3809ac40 2024-03-29T11:20:34.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.820+0000 7fde406d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:34.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.820+0000 7fde406d6640 10 monclient(hunting): authenticate will time out at 880.108533s 2024-03-29T11:20:34.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.820+0000 7fde3ec73640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fde380b4360 0x7fde3814d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:51508/0 (socket says 172.21.15.196:51508) 2024-03-29T11:20:34.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.820+0000 7fde3ec73640 1 -- 172.21.15.196:0/4224286500 learned_addr learned my addr 172.21.15.196:0/4224286500 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-29T11:20:34.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.820+0000 7fde3d470640 1 -- 172.21.15.196:0/4224286500 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2415039956 0 0) 0x7fde3809da30 con 0x7fde380b4360 2024-03-29T11:20:34.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.820+0000 7fde3d470640 10 monclient(hunting): _negotiate 2024-03-29T11:20:34.806 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.820+0000 7fde3d470640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:34.806 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.820+0000 7fde3d470640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:34.806 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.820+0000 7fde3d470640 10 monclient(hunting): my global_id is 4129 2024-03-29T11:20:34.806 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.820+0000 7fde3d470640 1 -- 172.21.15.196:0/4224286500 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde24003520 con 0x7fde380b4360 2024-03-29T11:20:34.806 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde3d470640 1 -- 172.21.15.196:0/4224286500 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3372021158 0 0) 0x7fde28001110 con 0x7fde3809c870 2024-03-29T11:20:34.806 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde3d470640 10 monclient(hunting): _negotiate 2024-03-29T11:20:34.806 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde3d470640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:34.806 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde3d470640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:34.806 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde3d470640 10 monclient(hunting): my global_id is 4131 2024-03-29T11:20:34.806 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde3d470640 1 -- 172.21.15.196:0/4224286500 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde3809da30 con 0x7fde3809c870 2024-03-29T11:20:34.806 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde3d470640 1 -- 172.21.15.196:0/4224286500 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3546785456 0 0) 0x7fde380b39c0 con 0x7fde3809ac40 2024-03-29T11:20:34.806 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde3d470640 10 monclient(hunting): _negotiate 2024-03-29T11:20:34.807 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde3d470640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:34.807 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde3d470640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:34.807 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde3d470640 10 monclient(hunting): my global_id is 4124 2024-03-29T11:20:34.807 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde3d470640 1 -- 172.21.15.196:0/4224286500 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde28001110 con 0x7fde3809ac40 2024-03-29T11:20:34.807 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde3d470640 1 -- 172.21.15.196:0/4224286500 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2700576443 0 0) 0x7fde24003520 con 0x7fde380b4360 2024-03-29T11:20:34.807 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde3d470640 1 -- 172.21.15.196:0/4224286500 >> v1:172.21.15.86:6790/0 conn(0x7fde3809ac40 legacy=0x7fde380b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.807 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde3d470640 1 -- 172.21.15.196:0/4224286500 >> v1:172.21.15.86:6789/0 conn(0x7fde3809c870 legacy=0x7fde3813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.807 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde3d470640 10 monclient: _finish_hunting 0 2024-03-29T11:20:34.807 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde3d470640 1 monclient: found mon.noname-c 2024-03-29T11:20:34.807 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde3d470640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:34.807 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde3d470640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T11:20:34.807 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde3d470640 1 -- 172.21.15.196:0/4224286500 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde380b5160 con 0x7fde380b4360 2024-03-29T11:20:34.807 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde3d470640 10 monclient: _finish_auth 0 2024-03-29T11:20:34.808 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde3d470640 10 monclient: _check_auth_tickets 2024-03-29T11:20:34.808 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde3d470640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:34.808 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde3d470640 1 -- 172.21.15.196:0/4224286500 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fde30001a10 con 0x7fde380b4360 2024-03-29T11:20:34.808 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde406d6640 5 monclient: authenticate success, global_id 4129 2024-03-29T11:20:34.808 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde406d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:34.808 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde3d470640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:34.808 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde3d470640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:34.808 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde3d470640 10 monclient: dump: 2024-03-29T11:20:34.808 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:34.808 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:34.808 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.808 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.808 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:34.808 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:34.808 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:34.809 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:34.809 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:34.809 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:34.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde406d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:34.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde3d470640 1 -- 172.21.15.196:0/4224286500 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde30001b70 con 0x7fde380b4360 2024-03-29T11:20:34.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde3d470640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:34.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde3d470640 1 -- 172.21.15.196:0/4224286500 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fde30001f80 con 0x7fde380b4360 2024-03-29T11:20:34.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde406d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:34.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde406d6640 4 set_mon_vals no callback set 2024-03-29T11:20:34.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde3d470640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:34.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde3d470640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:34.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde3d470640 10 monclient: dump: 2024-03-29T11:20:34.809 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:34.809 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:34.809 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.810 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.810 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:34.810 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:34.810 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:34.810 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:34.810 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:34.810 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:34.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde406d6640 10 monclient: shutdown 2024-03-29T11:20:34.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde406d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:34.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde406d6640 1 -- 172.21.15.196:0/4224286500 >> v1:172.21.15.196:6789/0 conn(0x7fde380b4360 legacy=0x7fde3814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde406d6640 1 -- 172.21.15.196:0/4224286500 shutdown_connections 2024-03-29T11:20:34.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde406d6640 1 -- 172.21.15.196:0/4224286500 >> 172.21.15.196:0/4224286500 conn(0x7fde38066b00 msgr2=0x7fde38066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:34.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde406d6640 1 -- 172.21.15.196:0/4224286500 shutdown_connections 2024-03-29T11:20:34.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.821+0000 7fde406d6640 1 -- 172.21.15.196:0/4224286500 wait complete. 2024-03-29T11:20:34.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.822+0000 7fde406d6640 10 monclient: build_initial_monmap 2024-03-29T11:20:34.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.822+0000 7fde406d6640 10 monclient: monmap: 2024-03-29T11:20:34.811 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-29T11:20:34.811 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:34.811 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 0.000000 2024-03-29T11:20:34.811 INFO:teuthology.orchestra.run.smithi196.stderr:created 0.000000 2024-03-29T11:20:34.811 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:34.811 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:34.811 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:20:34.811 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:20:34.811 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:20:34.811 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:34.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.822+0000 7fde406d6640 1 librados: starting msgr at 2024-03-29T11:20:34.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.822+0000 7fde406d6640 1 librados: starting objecter 2024-03-29T11:20:34.812 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.822+0000 7fde406d6640 1 Processor -- start 2024-03-29T11:20:34.812 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.822+0000 7fde406d6640 1 -- start start 2024-03-29T11:20:34.812 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.822+0000 7fde406d6640 1 librados: setting wanted keys 2024-03-29T11:20:34.812 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.822+0000 7fde406d6640 1 librados: calling monclient init 2024-03-29T11:20:34.812 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.822+0000 7fde406d6640 10 monclient: init 2024-03-29T11:20:34.812 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde406d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:34.812 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde406d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:20:34.812 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde406d6640 10 monclient(hunting): picked mon.noname-b con 0x7fde3809ac40 addr v1:172.21.15.196:6789/0 2024-03-29T11:20:34.812 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde406d6640 10 monclient(hunting): picked mon.noname-a con 0x7fde381d7cd0 addr v1:172.21.15.86:6789/0 2024-03-29T11:20:34.812 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde406d6640 10 monclient(hunting): picked mon.noname-c con 0x7fde380ad1e0 addr v1:172.21.15.86:6790/0 2024-03-29T11:20:34.812 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde406d6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde381d7130 con 0x7fde381d7cd0 2024-03-29T11:20:34.812 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde406d6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde38139b10 con 0x7fde3809ac40 2024-03-29T11:20:34.812 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde406d6640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde3813ffa0 con 0x7fde380ad1e0 2024-03-29T11:20:34.812 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde406d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:34.812 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde406d6640 10 monclient(hunting): authenticate will time out at 880.111558s 2024-03-29T11:20:34.812 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde3e472640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fde3809ac40 0x7fde381d75c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:51518/0 (socket says 172.21.15.196:51518) 2024-03-29T11:20:34.813 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde3e472640 1 -- 172.21.15.196:0/1309564022 learned_addr learned my addr 172.21.15.196:0/1309564022 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-29T11:20:34.813 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde2effd640 10 client.?.objecter ms_handle_connect 0x7fde3809ac40 2024-03-29T11:20:34.813 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde2effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:34.813 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde2effd640 10 client.?.objecter ms_handle_connect 0x7fde381d7cd0 2024-03-29T11:20:34.813 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde2effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:34.813 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde2effd640 10 client.?.objecter ms_handle_connect 0x7fde380ad1e0 2024-03-29T11:20:34.813 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde2effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:34.813 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde2effd640 1 -- 172.21.15.196:0/1309564022 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2184237133 0 0) 0x7fde38139b10 con 0x7fde3809ac40 2024-03-29T11:20:34.813 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde2effd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:34.813 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde2effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:34.813 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:34.813 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde2effd640 10 monclient(hunting): my global_id is 4132 2024-03-29T11:20:34.813 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde2effd640 1 -- 172.21.15.196:0/1309564022 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde1c004eb0 con 0x7fde3809ac40 2024-03-29T11:20:34.813 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde2effd640 1 -- 172.21.15.196:0/1309564022 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1476950319 0 0) 0x7fde381d7130 con 0x7fde381d7cd0 2024-03-29T11:20:34.813 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde2effd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:34.814 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde2effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:34.814 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:34.814 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde2effd640 10 monclient(hunting): my global_id is 4134 2024-03-29T11:20:34.814 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde2effd640 1 -- 172.21.15.196:0/1309564022 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde38139b10 con 0x7fde381d7cd0 2024-03-29T11:20:34.814 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde2effd640 1 -- 172.21.15.196:0/1309564022 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2242208387 0 0) 0x7fde3813ffa0 con 0x7fde380ad1e0 2024-03-29T11:20:34.814 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde2effd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:34.814 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde2effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:34.814 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:34.814 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde2effd640 10 monclient(hunting): my global_id is 4127 2024-03-29T11:20:34.814 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.823+0000 7fde2effd640 1 -- 172.21.15.196:0/1309564022 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde381d7130 con 0x7fde380ad1e0 2024-03-29T11:20:34.814 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 1 -- 172.21.15.196:0/1309564022 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2834960987 0 0) 0x7fde38139b10 con 0x7fde381d7cd0 2024-03-29T11:20:34.814 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 1 -- 172.21.15.196:0/1309564022 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde3813ffa0 con 0x7fde381d7cd0 2024-03-29T11:20:34.814 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 1 -- 172.21.15.196:0/1309564022 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2764588048 0 0) 0x7fde1c004eb0 con 0x7fde3809ac40 2024-03-29T11:20:34.814 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 1 -- 172.21.15.196:0/1309564022 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde38139b10 con 0x7fde3809ac40 2024-03-29T11:20:34.814 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 1 -- 172.21.15.196:0/1309564022 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fde280018e0 con 0x7fde381d7cd0 2024-03-29T11:20:34.815 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:34.815 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:34.815 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 10 monclient(hunting): dump: 2024-03-29T11:20:34.815 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:34.815 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:34.815 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.815 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.815 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:34.815 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:34.815 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:34.815 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:34.815 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:34.815 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:34.815 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:34.815 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 1 -- 172.21.15.196:0/1309564022 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fde34001930 con 0x7fde3809ac40 2024-03-29T11:20:34.815 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:34.816 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:34.816 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 10 monclient(hunting): dump: 2024-03-29T11:20:34.816 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:34.816 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:34.816 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.816 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.816 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:34.816 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:34.816 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:34.816 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:34.816 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:34.816 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:34.816 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 1 -- 172.21.15.196:0/1309564022 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2442436490 0 0) 0x7fde381d7130 con 0x7fde380ad1e0 2024-03-29T11:20:34.816 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 1 -- 172.21.15.196:0/1309564022 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fde1c004eb0 con 0x7fde380ad1e0 2024-03-29T11:20:34.817 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 1 -- 172.21.15.196:0/1309564022 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fde30001ce0 con 0x7fde380ad1e0 2024-03-29T11:20:34.817 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:34.817 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:34.817 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 10 monclient(hunting): dump: 2024-03-29T11:20:34.817 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:34.817 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:34.817 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.817 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.817 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:34.817 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:34.817 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:34.817 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:34.817 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:34.817 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:34.817 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 1 -- 172.21.15.196:0/1309564022 <== mon.0 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3634603508 0 0) 0x7fde3813ffa0 con 0x7fde381d7cd0 2024-03-29T11:20:34.818 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 1 -- 172.21.15.196:0/1309564022 >> v1:172.21.15.86:6790/0 conn(0x7fde380ad1e0 legacy=0x7fde38139400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.818 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 1 -- 172.21.15.196:0/1309564022 >> v1:172.21.15.196:6789/0 conn(0x7fde3809ac40 legacy=0x7fde381d75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.818 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:34.818 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 1 monclient: found mon.a 2024-03-29T11:20:34.818 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:34.818 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:20:34.818 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 1 -- 172.21.15.196:0/1309564022 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde380ae520 con 0x7fde381d7cd0 2024-03-29T11:20:34.818 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 10 monclient: _finish_auth 0 2024-03-29T11:20:34.818 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:34.818 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:34.818 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde406d6640 5 monclient: authenticate success, global_id 4134 2024-03-29T11:20:34.818 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde406d6640 10 monclient: _renew_subs 2024-03-29T11:20:34.818 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde406d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:20:34.818 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde406d6640 1 -- 172.21.15.196:0/1309564022 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde380ae800 con 0x7fde381d7cd0 2024-03-29T11:20:34.819 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde406d6640 10 client.4134.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:34.819 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde406d6640 10 monclient: _renew_subs 2024-03-29T11:20:34.819 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde406d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:20:34.819 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde406d6640 1 -- 172.21.15.196:0/1309564022 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fde38142b70 con 0x7fde381d7cd0 2024-03-29T11:20:34.819 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde406d6640 1 librados: init done 2024-03-29T11:20:34.819 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 1 -- 172.21.15.196:0/1309564022 <== mon.0 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fde28001a40 con 0x7fde381d7cd0 2024-03-29T11:20:34.819 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:34.819 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 1 -- 172.21.15.196:0/1309564022 <== mon.0 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fde28003810 con 0x7fde381d7cd0 2024-03-29T11:20:34.819 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:34.819 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:34.819 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 10 monclient: dump: 2024-03-29T11:20:34.819 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:34.819 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:34.819 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.819 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:34.819 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:34.820 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:34.820 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:34.820 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:34.820 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:34.820 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:34.820 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde3cc6f640 4 set_mon_vals no callback set 2024-03-29T11:20:34.820 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 1 -- 172.21.15.196:0/1309564022 <== mon.0 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fde28003e00 con 0x7fde381d7cd0 2024-03-29T11:20:34.820 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 1 -- 172.21.15.196:0/1309564022 <== mon.0 v1:172.21.15.86:6789/0 8 ==== osd_map(5..5 src has 1..5) v4 ==== 1070+0+0 (unknown 15530546 0 0) 0x7fde28004540 con 0x7fde381d7cd0 2024-03-29T11:20:34.820 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 10 client.4134.objecter ms_dispatch 0x7fde38000b90 osd_map(5..5 src has 1..5) v4 2024-03-29T11:20:34.820 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 3 client.4134.objecter handle_osd_map got epochs [5,5] > 0 2024-03-29T11:20:34.820 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 3 client.4134.objecter handle_osd_map decoding full epoch 5 2024-03-29T11:20:34.820 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.824+0000 7fde2effd640 20 client.4134.objecter dump_active .. 0 homeless 2024-03-29T11:20:34.820 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.825+0000 7fde406d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:34.820 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.825+0000 7fde406d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:34.821 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.825+0000 7fde406d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:20:34.821 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.825+0000 7fde406d6640 1 -- 172.21.15.196:0/1309564022 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde20005080 con 0x7fde381d7cd0 2024-03-29T11:20:34.821 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.827+0000 7fde2effd640 1 -- 172.21.15.196:0/1309564022 <== mon.0 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fde28004160 con 0x7fde381d7cd0 2024-03-29T11:20:34.821 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.827+0000 7fde2effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:34.821 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.827+0000 7fde2effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:34.821 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T11:20:34.837+0000 7fee3a432040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-29T11:20:34.874 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.897+0000 7fde406d6640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "a4113954-1342-43e6-8807-bddf6ae9d460", "id": 3}] 2024-03-29T11:20:34.874 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.897+0000 7fde406d6640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "a4113954-1342-43e6-8807-bddf6ae9d460", "id": 3}] 2024-03-29T11:20:34.874 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.897+0000 7fde406d6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:20:34.875 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.897+0000 7fde406d6640 1 -- 172.21.15.196:0/1309564022 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd new", "uuid": "a4113954-1342-43e6-8807-bddf6ae9d460", "id": 3} v 0) v1 -- 0x7fde200058b0 con 0x7fde381d7cd0 2024-03-29T11:20:34.875 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.899+0000 7fde2effd640 1 -- 172.21.15.196:0/1309564022 <== mon.0 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "a4113954-1342-43e6-8807-bddf6ae9d460", "id": 3}]=0 v6) v1 ==== 112+0+2 (unknown 276038158 0 525046306) 0x7fde28004850 con 0x7fde381d7cd0 2024-03-29T11:20:34.876 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.899+0000 7fde2effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "a4113954-1342-43e6-8807-bddf6ae9d460", "id": 3}] 2024-03-29T11:20:34.876 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.899+0000 7fde2effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:34.876 INFO:teuthology.orchestra.run.smithi196.stdout:3 2024-03-29T11:20:34.877 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.900+0000 7fde406d6640 10 librados: watch_flush enter 2024-03-29T11:20:34.877 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.900+0000 7fde406d6640 10 librados: watch_flush exit 2024-03-29T11:20:34.877 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.900+0000 7fde406d6640 20 client.4134.objecter shutdown clearing up homeless session... 2024-03-29T11:20:34.877 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.900+0000 7fde406d6640 10 client.4134.objecter successfully canceled tick 2024-03-29T11:20:34.877 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.900+0000 7fde406d6640 10 monclient: shutdown 2024-03-29T11:20:34.877 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.900+0000 7fde406d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:34.877 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.900+0000 7fde406d6640 1 -- 172.21.15.196:0/1309564022 >> v1:172.21.15.86:6789/0 conn(0x7fde381d7cd0 legacy=0x7fde38138cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:34.878 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.900+0000 7fde406d6640 1 -- 172.21.15.196:0/1309564022 shutdown_connections 2024-03-29T11:20:34.878 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.900+0000 7fde406d6640 1 -- 172.21.15.196:0/1309564022 >> 172.21.15.196:0/1309564022 conn(0x7fde38066b00 msgr2=0x7fde380b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:34.878 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.900+0000 7fde406d6640 1 -- 172.21.15.196:0/1309564022 shutdown_connections 2024-03-29T11:20:34.878 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.900+0000 7fde406d6640 1 -- 172.21.15.196:0/1309564022 wait complete. 2024-03-29T11:20:34.878 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:34.900+0000 7fde406d6640 1 librados: shutdown 2024-03-29T11:20:34.887 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 04a3b66e-d800-443e-ade7-184a154ee5a3 4 2024-03-29T11:20:34.897 INFO:tasks.ceph.mgr.y.smithi086.stderr:2024-03-29T11:20:34.920+0000 7f4bd1e42040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:20:34.933 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T11:20:34.956+0000 7fee3a432040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-29T11:20:34.967 INFO:tasks.ceph.mgr.y.smithi086.stderr:2024-03-29T11:20:34.991+0000 7f4bd1e42040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:20:35.006 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T11:20:35.030+0000 7fee3a432040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-29T11:20:35.013 INFO:tasks.ceph.mgr.y.smithi086.stderr:2024-03-29T11:20:35.037+0000 7f4bd1e42040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:20:35.025 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.049+0000 7f58cad23640 10 monclient: get_monmap_and_config 2024-03-29T11:20:35.026 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.049+0000 7f58cad23640 10 monclient: build_initial_monmap 2024-03-29T11:20:35.026 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.049+0000 7f58cad23640 10 monclient: monmap: 2024-03-29T11:20:35.026 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-29T11:20:35.026 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:35.026 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:20:35.050250+0000 2024-03-29T11:20:35.026 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:20:35.050250+0000 2024-03-29T11:20:35.026 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:35.026 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:35.026 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:20:35.026 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:20:35.026 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:20:35.026 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:35.034 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.049+0000 7f58cad23640 1 Processor -- start 2024-03-29T11:20:35.034 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.049+0000 7f58cad23640 1 -- start start 2024-03-29T11:20:35.034 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.049+0000 7f58cad23640 10 monclient: init 2024-03-29T11:20:35.035 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.050+0000 7f58cad23640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:35.035 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.050+0000 7f58cad23640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:35.035 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.050+0000 7f58cad23640 10 monclient(hunting): picked mon.noname-b con 0x7f58c409ac40 addr v1:172.21.15.86:6790/0 2024-03-29T11:20:35.035 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.050+0000 7f58cad23640 10 monclient(hunting): picked mon.noname-a con 0x7f58c409c870 addr v1:172.21.15.86:6789/0 2024-03-29T11:20:35.035 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.050+0000 7f58cad23640 10 monclient(hunting): picked mon.noname-c con 0x7f58c40b4360 addr v1:172.21.15.196:6789/0 2024-03-29T11:20:35.035 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.050+0000 7f58cad23640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c409dec0 con 0x7f58c409c870 2024-03-29T11:20:35.035 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.050+0000 7f58cad23640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c409da30 con 0x7f58c40b4360 2024-03-29T11:20:35.035 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.050+0000 7f58cad23640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c40b39c0 con 0x7f58c409ac40 2024-03-29T11:20:35.035 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.050+0000 7f58cad23640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:35.035 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.050+0000 7f58cad23640 10 monclient(hunting): authenticate will time out at 880.338539s 2024-03-29T11:20:35.035 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.050+0000 7f58c92c0640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f58c40b4360 0x7f58c414d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:51520/0 (socket says 172.21.15.196:51520) 2024-03-29T11:20:35.035 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.050+0000 7f58c92c0640 1 -- 172.21.15.196:0/2529446607 learned_addr learned my addr 172.21.15.196:0/2529446607 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-29T11:20:35.035 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.050+0000 7f58c37fe640 1 -- 172.21.15.196:0/2529446607 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3385053602 0 0) 0x7f58c409da30 con 0x7f58c40b4360 2024-03-29T11:20:35.035 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.050+0000 7f58c37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.036 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.050+0000 7f58c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.036 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.050+0000 7f58c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.036 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.050+0000 7f58c37fe640 10 monclient(hunting): my global_id is 4135 2024-03-29T11:20:35.036 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.050+0000 7f58c37fe640 1 -- 172.21.15.196:0/2529446607 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58b4003520 con 0x7f58c40b4360 2024-03-29T11:20:35.036 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.050+0000 7f58c37fe640 1 -- 172.21.15.196:0/2529446607 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3426702775 0 0) 0x7f58b0001110 con 0x7f58c409c870 2024-03-29T11:20:35.036 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.050+0000 7f58c37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.036 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.050+0000 7f58c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.036 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.050+0000 7f58c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.036 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.050+0000 7f58c37fe640 10 monclient(hunting): my global_id is 4137 2024-03-29T11:20:35.036 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.050+0000 7f58c37fe640 1 -- 172.21.15.196:0/2529446607 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58c409da30 con 0x7f58c409c870 2024-03-29T11:20:35.036 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.050+0000 7f58c37fe640 1 -- 172.21.15.196:0/2529446607 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 834777803 0 0) 0x7f58c40b39c0 con 0x7f58c409ac40 2024-03-29T11:20:35.036 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58c37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.036 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.036 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58c37fe640 10 monclient(hunting): my global_id is 4130 2024-03-29T11:20:35.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58c37fe640 1 -- 172.21.15.196:0/2529446607 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58b0001110 con 0x7f58c409ac40 2024-03-29T11:20:35.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58c37fe640 1 -- 172.21.15.196:0/2529446607 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2269981748 0 0) 0x7f58b4003520 con 0x7f58c40b4360 2024-03-29T11:20:35.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58c37fe640 1 -- 172.21.15.196:0/2529446607 >> v1:172.21.15.86:6790/0 conn(0x7f58c409ac40 legacy=0x7f58c40b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58c37fe640 1 -- 172.21.15.196:0/2529446607 >> v1:172.21.15.86:6789/0 conn(0x7f58c409c870 legacy=0x7f58c413f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58c37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:20:35.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58c37fe640 1 monclient: found mon.noname-c 2024-03-29T11:20:35.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:35.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58c37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T11:20:35.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58c37fe640 1 -- 172.21.15.196:0/2529446607 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c40b5160 con 0x7f58c40b4360 2024-03-29T11:20:35.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58c37fe640 10 monclient: _finish_auth 0 2024-03-29T11:20:35.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58c37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:20:35.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:35.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58c37fe640 1 -- 172.21.15.196:0/2529446607 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f58bc001a10 con 0x7f58c40b4360 2024-03-29T11:20:35.037 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58cad23640 5 monclient: authenticate success, global_id 4135 2024-03-29T11:20:35.038 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58cad23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:35.038 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.038 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58c37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:35.038 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58c37fe640 10 monclient: dump: 2024-03-29T11:20:35.038 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:35.038 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:35.038 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.038 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.038 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.038 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:35.038 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:35.038 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:35.038 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:35.038 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:35.038 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58cad23640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:35.039 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58c37fe640 1 -- 172.21.15.196:0/2529446607 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58bc001bf0 con 0x7f58c40b4360 2024-03-29T11:20:35.039 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58c37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:35.039 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58cad23640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:35.039 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58cad23640 4 set_mon_vals no callback set 2024-03-29T11:20:35.039 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58c37fe640 1 -- 172.21.15.196:0/2529446607 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f58bc002000 con 0x7f58c40b4360 2024-03-29T11:20:35.039 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.039 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58c37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:35.039 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58c37fe640 10 monclient: dump: 2024-03-29T11:20:35.039 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:35.039 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:35.039 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.039 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.039 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.039 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:35.039 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:35.040 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:35.040 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:35.040 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:35.040 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58cad23640 10 monclient: shutdown 2024-03-29T11:20:35.040 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58cad23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:35.040 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58cad23640 1 -- 172.21.15.196:0/2529446607 >> v1:172.21.15.196:6789/0 conn(0x7f58c40b4360 legacy=0x7f58c414d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.040 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58cad23640 1 -- 172.21.15.196:0/2529446607 shutdown_connections 2024-03-29T11:20:35.040 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58cad23640 1 -- 172.21.15.196:0/2529446607 >> 172.21.15.196:0/2529446607 conn(0x7f58c4066b00 msgr2=0x7f58c4066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:35.040 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58cad23640 1 -- 172.21.15.196:0/2529446607 shutdown_connections 2024-03-29T11:20:35.040 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.051+0000 7f58cad23640 1 -- 172.21.15.196:0/2529446607 wait complete. 2024-03-29T11:20:35.040 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.052+0000 7f58cad23640 10 monclient: build_initial_monmap 2024-03-29T11:20:35.040 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.052+0000 7f58cad23640 10 monclient: monmap: 2024-03-29T11:20:35.040 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-29T11:20:35.040 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:35.040 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 0.000000 2024-03-29T11:20:35.041 INFO:teuthology.orchestra.run.smithi196.stderr:created 0.000000 2024-03-29T11:20:35.041 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:35.041 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:35.041 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:20:35.041 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:20:35.041 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:20:35.041 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:35.041 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.052+0000 7f58cad23640 1 librados: starting msgr at 2024-03-29T11:20:35.041 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.052+0000 7f58cad23640 1 librados: starting objecter 2024-03-29T11:20:35.041 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.052+0000 7f58cad23640 1 Processor -- start 2024-03-29T11:20:35.041 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.052+0000 7f58cad23640 1 -- start start 2024-03-29T11:20:35.041 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.052+0000 7f58cad23640 1 librados: setting wanted keys 2024-03-29T11:20:35.041 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.052+0000 7f58cad23640 1 librados: calling monclient init 2024-03-29T11:20:35.041 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.052+0000 7f58cad23640 10 monclient: init 2024-03-29T11:20:35.041 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.053+0000 7f58cad23640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:35.042 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.053+0000 7f58cad23640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:35.042 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.053+0000 7f58cad23640 10 monclient(hunting): picked mon.noname-b con 0x7f58c409ac40 addr v1:172.21.15.196:6789/0 2024-03-29T11:20:35.042 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.053+0000 7f58cad23640 10 monclient(hunting): picked mon.noname-c con 0x7f58c41d7cd0 addr v1:172.21.15.86:6790/0 2024-03-29T11:20:35.042 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.053+0000 7f58cad23640 10 monclient(hunting): picked mon.noname-a con 0x7f58c40ad1e0 addr v1:172.21.15.86:6789/0 2024-03-29T11:20:35.042 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.053+0000 7f58cad23640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c41d7130 con 0x7f58c40ad1e0 2024-03-29T11:20:35.042 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.053+0000 7f58cad23640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c4139b10 con 0x7f58c409ac40 2024-03-29T11:20:35.042 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.053+0000 7f58cad23640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f58c413ffa0 con 0x7f58c41d7cd0 2024-03-29T11:20:35.042 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.053+0000 7f58cad23640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:35.042 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.053+0000 7f58cad23640 10 monclient(hunting): authenticate will time out at 880.341631s 2024-03-29T11:20:35.042 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.053+0000 7f58c8abf640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f58c409ac40 0x7f58c41d75c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:51522/0 (socket says 172.21.15.196:51522) 2024-03-29T11:20:35.042 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.053+0000 7f58c8abf640 1 -- 172.21.15.196:0/3033376522 learned_addr learned my addr 172.21.15.196:0/3033376522 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-29T11:20:35.042 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.053+0000 7f58c17fa640 10 client.?.objecter ms_handle_connect 0x7f58c409ac40 2024-03-29T11:20:35.042 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.053+0000 7f58c17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:35.042 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.053+0000 7f58c17fa640 10 client.?.objecter ms_handle_connect 0x7f58c41d7cd0 2024-03-29T11:20:35.043 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.053+0000 7f58c17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:35.043 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.053+0000 7f58c17fa640 10 client.?.objecter ms_handle_connect 0x7f58c40ad1e0 2024-03-29T11:20:35.043 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.053+0000 7f58c17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:35.043 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.053+0000 7f58c17fa640 1 -- 172.21.15.196:0/3033376522 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1510479406 0 0) 0x7f58c4139b10 con 0x7f58c409ac40 2024-03-29T11:20:35.043 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.053+0000 7f58c17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.043 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.053+0000 7f58c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.043 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.053+0000 7f58c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.043 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.053+0000 7f58c17fa640 10 monclient(hunting): my global_id is 4138 2024-03-29T11:20:35.043 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.053+0000 7f58c17fa640 1 -- 172.21.15.196:0/3033376522 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58a8004eb0 con 0x7f58c409ac40 2024-03-29T11:20:35.043 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.053+0000 7f58c17fa640 1 -- 172.21.15.196:0/3033376522 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1434019695 0 0) 0x7f58c413ffa0 con 0x7f58c41d7cd0 2024-03-29T11:20:35.043 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.053+0000 7f58c17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.043 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.053+0000 7f58c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.043 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.053+0000 7f58c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.043 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.053+0000 7f58c17fa640 10 monclient(hunting): my global_id is 4133 2024-03-29T11:20:35.043 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 1 -- 172.21.15.196:0/3033376522 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58c4139b10 con 0x7f58c41d7cd0 2024-03-29T11:20:35.044 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 1 -- 172.21.15.196:0/3033376522 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1412258836 0 0) 0x7f58c41d7130 con 0x7f58c40ad1e0 2024-03-29T11:20:35.044 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.044 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.044 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.044 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 10 monclient(hunting): my global_id is 4140 2024-03-29T11:20:35.044 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 1 -- 172.21.15.196:0/3033376522 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f58c413ffa0 con 0x7f58c40ad1e0 2024-03-29T11:20:35.044 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 1 -- 172.21.15.196:0/3033376522 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 820798757 0 0) 0x7f58a8004eb0 con 0x7f58c409ac40 2024-03-29T11:20:35.044 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 1 -- 172.21.15.196:0/3033376522 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58c41d7130 con 0x7f58c409ac40 2024-03-29T11:20:35.044 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 1 -- 172.21.15.196:0/3033376522 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f58b8001a10 con 0x7f58c409ac40 2024-03-29T11:20:35.045 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.045 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:35.045 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 10 monclient(hunting): dump: 2024-03-29T11:20:35.045 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:35.045 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:35.045 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.045 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.045 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.045 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:35.045 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:35.045 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:35.045 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:35.045 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:35.045 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:35.046 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 1 -- 172.21.15.196:0/3033376522 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2672007531 0 0) 0x7f58c4139b10 con 0x7f58c41d7cd0 2024-03-29T11:20:35.046 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 1 -- 172.21.15.196:0/3033376522 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58a8004eb0 con 0x7f58c41d7cd0 2024-03-29T11:20:35.046 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 1 -- 172.21.15.196:0/3033376522 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f58b0001920 con 0x7f58c41d7cd0 2024-03-29T11:20:35.046 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.046 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:35.046 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 10 monclient(hunting): dump: 2024-03-29T11:20:35.046 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:35.046 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:35.046 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.046 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.046 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.046 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:35.046 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:35.046 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:35.047 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:35.047 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:35.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 1 -- 172.21.15.196:0/3033376522 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1744261230 0 0) 0x7f58c413ffa0 con 0x7f58c40ad1e0 2024-03-29T11:20:35.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 1 -- 172.21.15.196:0/3033376522 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f58c4139b10 con 0x7f58c40ad1e0 2024-03-29T11:20:35.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 1 -- 172.21.15.196:0/3033376522 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f58bc001d60 con 0x7f58c40ad1e0 2024-03-29T11:20:35.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:35.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 10 monclient(hunting): dump: 2024-03-29T11:20:35.047 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:35.047 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:35.047 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.047 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.047 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.047 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:35.048 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:35.048 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:35.048 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:35.048 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:35.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 1 -- 172.21.15.196:0/3033376522 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 457863593 0 0) 0x7f58c41d7130 con 0x7f58c409ac40 2024-03-29T11:20:35.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 1 -- 172.21.15.196:0/3033376522 >> v1:172.21.15.86:6790/0 conn(0x7f58c41d7cd0 legacy=0x7f58c4138cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 1 -- 172.21.15.196:0/3033376522 >> v1:172.21.15.86:6789/0 conn(0x7f58c40ad1e0 legacy=0x7f58c4139400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:20:35.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 1 monclient: found mon.b 2024-03-29T11:20:35.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:35.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:20:35.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 1 -- 172.21.15.196:0/3033376522 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58c40ae520 con 0x7f58c409ac40 2024-03-29T11:20:35.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 10 monclient: _finish_auth 0 2024-03-29T11:20:35.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:20:35.049 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:35.049 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58cad23640 5 monclient: authenticate success, global_id 4138 2024-03-29T11:20:35.049 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58cad23640 10 monclient: _renew_subs 2024-03-29T11:20:35.049 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58cad23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:20:35.049 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58cad23640 1 -- 172.21.15.196:0/3033376522 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58c40ae800 con 0x7f58c409ac40 2024-03-29T11:20:35.049 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58cad23640 10 client.4138.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:35.049 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58cad23640 10 monclient: _renew_subs 2024-03-29T11:20:35.049 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58cad23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:20:35.049 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58cad23640 1 -- 172.21.15.196:0/3033376522 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f58c4142b70 con 0x7f58c409ac40 2024-03-29T11:20:35.049 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58cad23640 1 librados: init done 2024-03-29T11:20:35.049 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 1 -- 172.21.15.196:0/3033376522 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f58b8001c70 con 0x7f58c409ac40 2024-03-29T11:20:35.049 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:35.049 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 1 -- 172.21.15.196:0/3033376522 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f58b8003a60 con 0x7f58c409ac40 2024-03-29T11:20:35.049 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.049 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:35.050 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 10 monclient: dump: 2024-03-29T11:20:35.050 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:35.050 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:35.050 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.050 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.050 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.050 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:35.050 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:35.050 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:35.050 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:35.050 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:35.050 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c2ffd640 4 set_mon_vals no callback set 2024-03-29T11:20:35.050 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 1 -- 172.21.15.196:0/3033376522 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f58b8004010 con 0x7f58c409ac40 2024-03-29T11:20:35.050 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 1 -- 172.21.15.196:0/3033376522 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(6..6 src has 1..6) v4 ==== 1185+0+0 (unknown 2180576754 0 0) 0x7f58b80068a0 con 0x7f58c409ac40 2024-03-29T11:20:35.051 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 10 client.4138.objecter ms_dispatch 0x7f58c4000b90 osd_map(6..6 src has 1..6) v4 2024-03-29T11:20:35.051 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 3 client.4138.objecter handle_osd_map got epochs [6,6] > 0 2024-03-29T11:20:35.051 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 3 client.4138.objecter handle_osd_map decoding full epoch 6 2024-03-29T11:20:35.051 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.054+0000 7f58c17fa640 20 client.4138.objecter dump_active .. 0 homeless 2024-03-29T11:20:35.051 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.055+0000 7f58cad23640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:35.051 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.055+0000 7f58cad23640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:35.051 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.055+0000 7f58cad23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:20:35.051 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.055+0000 7f58cad23640 1 -- 172.21.15.196:0/3033376522 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58a4005080 con 0x7f58c409ac40 2024-03-29T11:20:35.051 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.057+0000 7f58c17fa640 1 -- 172.21.15.196:0/3033376522 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f58b8006390 con 0x7f58c409ac40 2024-03-29T11:20:35.051 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.057+0000 7f58c17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:35.051 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.057+0000 7f58c17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:35.056 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T11:20:35.079+0000 7fee3a432040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-29T11:20:35.067 INFO:tasks.ceph.mgr.y.smithi086.stderr:2024-03-29T11:20:35.091+0000 7f4bd1e42040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:20:35.106 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.129+0000 7f58cad23640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "04a3b66e-d800-443e-ade7-184a154ee5a3", "id": 4}] 2024-03-29T11:20:35.106 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.129+0000 7f58cad23640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "04a3b66e-d800-443e-ade7-184a154ee5a3", "id": 4}] 2024-03-29T11:20:35.106 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.129+0000 7f58cad23640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:20:35.106 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.129+0000 7f58cad23640 1 -- 172.21.15.196:0/3033376522 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd new", "uuid": "04a3b66e-d800-443e-ade7-184a154ee5a3", "id": 4} v 0) v1 -- 0x7f58a40058b0 con 0x7f58c409ac40 2024-03-29T11:20:35.109 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.133+0000 7f58c17fa640 1 -- 172.21.15.196:0/3033376522 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "04a3b66e-d800-443e-ade7-184a154ee5a3", "id": 4}]=0 v7) v1 ==== 112+0+2 (unknown 3751362066 0 1697012583) 0x7f58b8004450 con 0x7f58c409ac40 2024-03-29T11:20:35.109 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.133+0000 7f58c17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "04a3b66e-d800-443e-ade7-184a154ee5a3", "id": 4}] 2024-03-29T11:20:35.109 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.133+0000 7f58c17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:35.110 INFO:teuthology.orchestra.run.smithi196.stdout:4 2024-03-29T11:20:35.110 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T11:20:35.133+0000 7fee3a432040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-29T11:20:35.111 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.133+0000 7f58cad23640 10 librados: watch_flush enter 2024-03-29T11:20:35.111 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.134+0000 7f58cad23640 10 librados: watch_flush exit 2024-03-29T11:20:35.111 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.134+0000 7f58cad23640 20 client.4138.objecter shutdown clearing up homeless session... 2024-03-29T11:20:35.111 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.134+0000 7f58cad23640 10 client.4138.objecter successfully canceled tick 2024-03-29T11:20:35.111 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.134+0000 7f58cad23640 10 monclient: shutdown 2024-03-29T11:20:35.111 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.134+0000 7f58cad23640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:35.112 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.134+0000 7f58cad23640 1 -- 172.21.15.196:0/3033376522 >> v1:172.21.15.196:6789/0 conn(0x7f58c409ac40 legacy=0x7f58c41d75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.112 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.134+0000 7f58cad23640 1 -- 172.21.15.196:0/3033376522 shutdown_connections 2024-03-29T11:20:35.112 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.134+0000 7f58cad23640 1 -- 172.21.15.196:0/3033376522 >> 172.21.15.196:0/3033376522 conn(0x7f58c4066b00 msgr2=0x7f58c40b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:35.112 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.134+0000 7f58cad23640 1 -- 172.21.15.196:0/3033376522 shutdown_connections 2024-03-29T11:20:35.112 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.134+0000 7f58cad23640 1 -- 172.21.15.196:0/3033376522 wait complete. 2024-03-29T11:20:35.112 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.134+0000 7f58cad23640 1 librados: shutdown 2024-03-29T11:20:35.115 INFO:tasks.ceph.mgr.y.smithi086.stderr:2024-03-29T11:20:35.139+0000 7f4bd1e42040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:20:35.121 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 3ed0352b-96f6-4668-88dd-4747f5385eb4 5 2024-03-29T11:20:35.159 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T11:20:35.182+0000 7fee3a432040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-29T11:20:35.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.281+0000 7fcf557ba640 10 monclient: get_monmap_and_config 2024-03-29T11:20:35.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.281+0000 7fcf557ba640 10 monclient: build_initial_monmap 2024-03-29T11:20:35.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.281+0000 7fcf557ba640 10 monclient: monmap: 2024-03-29T11:20:35.259 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-29T11:20:35.259 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:35.259 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:20:35.283019+0000 2024-03-29T11:20:35.259 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:20:35.283019+0000 2024-03-29T11:20:35.259 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:35.259 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:35.259 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:20:35.259 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:20:35.259 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:20:35.260 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:35.260 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.282+0000 7fcf557ba640 1 Processor -- start 2024-03-29T11:20:35.260 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.282+0000 7fcf557ba640 1 -- start start 2024-03-29T11:20:35.260 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.282+0000 7fcf557ba640 10 monclient: init 2024-03-29T11:20:35.268 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf557ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:35.268 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf557ba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:35.268 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf557ba640 10 monclient(hunting): picked mon.noname-c con 0x7fcf5009ac40 addr v1:172.21.15.196:6789/0 2024-03-29T11:20:35.268 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf557ba640 10 monclient(hunting): picked mon.noname-b con 0x7fcf5009c870 addr v1:172.21.15.86:6790/0 2024-03-29T11:20:35.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf557ba640 10 monclient(hunting): picked mon.noname-a con 0x7fcf500b4360 addr v1:172.21.15.86:6789/0 2024-03-29T11:20:35.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf557ba640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf5009dec0 con 0x7fcf500b4360 2024-03-29T11:20:35.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf557ba640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf5009da30 con 0x7fcf5009ac40 2024-03-29T11:20:35.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf557ba640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf500b39c0 con 0x7fcf5009c870 2024-03-29T11:20:35.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf557ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:35.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf557ba640 10 monclient(hunting): authenticate will time out at 880.571218s 2024-03-29T11:20:35.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4effd640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fcf5009ac40 0x7fcf500b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:51538/0 (socket says 172.21.15.196:51538) 2024-03-29T11:20:35.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4effd640 1 -- 172.21.15.196:0/3307009134 learned_addr learned my addr 172.21.15.196:0/3307009134 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-29T11:20:35.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 1 -- 172.21.15.196:0/3307009134 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2628350991 0 0) 0x7fcf5009da30 con 0x7fcf5009ac40 2024-03-29T11:20:35.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 10 monclient(hunting): my global_id is 4141 2024-03-29T11:20:35.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 1 -- 172.21.15.196:0/3307009134 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf38003520 con 0x7fcf5009ac40 2024-03-29T11:20:35.269 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 1 -- 172.21.15.196:0/3307009134 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3104681245 0 0) 0x7fcf500b39c0 con 0x7fcf5009c870 2024-03-29T11:20:35.270 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.270 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.270 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.270 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 10 monclient(hunting): my global_id is 4136 2024-03-29T11:20:35.270 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 1 -- 172.21.15.196:0/3307009134 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf5009da30 con 0x7fcf5009c870 2024-03-29T11:20:35.270 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 1 -- 172.21.15.196:0/3307009134 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2559963679 0 0) 0x7fcf480013c0 con 0x7fcf500b4360 2024-03-29T11:20:35.270 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.270 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.270 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.270 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 10 monclient(hunting): my global_id is 4143 2024-03-29T11:20:35.270 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 1 -- 172.21.15.196:0/3307009134 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf500b39c0 con 0x7fcf500b4360 2024-03-29T11:20:35.270 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 1 -- 172.21.15.196:0/3307009134 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 862063122 0 0) 0x7fcf38003520 con 0x7fcf5009ac40 2024-03-29T11:20:35.270 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 1 -- 172.21.15.196:0/3307009134 >> v1:172.21.15.86:6790/0 conn(0x7fcf5009c870 legacy=0x7fcf5013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.270 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 1 -- 172.21.15.196:0/3307009134 >> v1:172.21.15.86:6789/0 conn(0x7fcf500b4360 legacy=0x7fcf5014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.270 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:20:35.271 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 1 monclient: found mon.noname-c 2024-03-29T11:20:35.271 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:35.271 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T11:20:35.273 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 1 -- 172.21.15.196:0/3307009134 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf500b5160 con 0x7fcf5009ac40 2024-03-29T11:20:35.273 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 10 monclient: _finish_auth 0 2024-03-29T11:20:35.273 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:20:35.273 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:35.274 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 1 -- 172.21.15.196:0/3307009134 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fcf44001690 con 0x7fcf5009ac40 2024-03-29T11:20:35.274 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf557ba640 5 monclient: authenticate success, global_id 4141 2024-03-29T11:20:35.274 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf557ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:35.274 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.274 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:35.274 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 10 monclient: dump: 2024-03-29T11:20:35.274 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:35.274 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:35.274 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.274 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.274 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.274 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:35.274 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:35.274 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:35.274 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:35.275 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:35.275 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf557ba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:35.275 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 1 -- 172.21.15.196:0/3307009134 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf44001870 con 0x7fcf5009ac40 2024-03-29T11:20:35.275 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:35.275 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf557ba640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:35.275 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 1 -- 172.21.15.196:0/3307009134 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fcf44001c80 con 0x7fcf5009ac40 2024-03-29T11:20:35.275 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf557ba640 4 set_mon_vals no callback set 2024-03-29T11:20:35.275 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.275 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:35.275 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.283+0000 7fcf4dffb640 10 monclient: dump: 2024-03-29T11:20:35.275 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:35.275 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:35.275 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.275 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.276 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.276 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:35.276 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:35.276 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:35.276 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:35.276 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:35.276 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.284+0000 7fcf557ba640 10 monclient: shutdown 2024-03-29T11:20:35.276 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.284+0000 7fcf557ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:35.276 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.284+0000 7fcf557ba640 1 -- 172.21.15.196:0/3307009134 >> v1:172.21.15.196:6789/0 conn(0x7fcf5009ac40 legacy=0x7fcf500b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.276 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.284+0000 7fcf557ba640 1 -- 172.21.15.196:0/3307009134 shutdown_connections 2024-03-29T11:20:35.276 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.284+0000 7fcf557ba640 1 -- 172.21.15.196:0/3307009134 >> 172.21.15.196:0/3307009134 conn(0x7fcf50066b00 msgr2=0x7fcf50066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:35.276 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.284+0000 7fcf557ba640 1 -- 172.21.15.196:0/3307009134 shutdown_connections 2024-03-29T11:20:35.276 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.284+0000 7fcf557ba640 1 -- 172.21.15.196:0/3307009134 wait complete. 2024-03-29T11:20:35.276 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.285+0000 7fcf557ba640 10 monclient: build_initial_monmap 2024-03-29T11:20:35.276 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.285+0000 7fcf557ba640 10 monclient: monmap: 2024-03-29T11:20:35.277 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-29T11:20:35.277 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:35.277 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 0.000000 2024-03-29T11:20:35.277 INFO:teuthology.orchestra.run.smithi196.stderr:created 0.000000 2024-03-29T11:20:35.277 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:35.277 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:35.277 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:20:35.277 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:20:35.277 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:20:35.277 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:35.277 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.285+0000 7fcf557ba640 1 librados: starting msgr at 2024-03-29T11:20:35.277 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.285+0000 7fcf557ba640 1 librados: starting objecter 2024-03-29T11:20:35.277 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.285+0000 7fcf557ba640 1 Processor -- start 2024-03-29T11:20:35.277 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.285+0000 7fcf557ba640 1 -- start start 2024-03-29T11:20:35.278 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.285+0000 7fcf557ba640 1 librados: setting wanted keys 2024-03-29T11:20:35.278 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.285+0000 7fcf557ba640 1 librados: calling monclient init 2024-03-29T11:20:35.278 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.285+0000 7fcf557ba640 10 monclient: init 2024-03-29T11:20:35.278 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf557ba640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:35.278 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf557ba640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:35.278 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf557ba640 10 monclient(hunting): picked mon.noname-c con 0x7fcf5009ac40 addr v1:172.21.15.86:6790/0 2024-03-29T11:20:35.278 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf557ba640 10 monclient(hunting): picked mon.noname-a con 0x7fcf501d7220 addr v1:172.21.15.86:6789/0 2024-03-29T11:20:35.278 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf557ba640 10 monclient(hunting): picked mon.noname-b con 0x7fcf501dfb50 addr v1:172.21.15.196:6789/0 2024-03-29T11:20:35.278 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf557ba640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf50142eb0 con 0x7fcf501d7220 2024-03-29T11:20:35.278 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf557ba640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf500b7700 con 0x7fcf501dfb50 2024-03-29T11:20:35.278 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf557ba640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf500b7970 con 0x7fcf5009ac40 2024-03-29T11:20:35.278 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf557ba640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:35.278 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf557ba640 10 monclient(hunting): authenticate will time out at 880.574211s 2024-03-29T11:20:35.278 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf4f7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fcf501dfb50 0x7fcf501e1f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:51546/0 (socket says 172.21.15.196:51546) 2024-03-29T11:20:35.279 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf4f7fe640 1 -- 172.21.15.196:0/2380061845 learned_addr learned my addr 172.21.15.196:0/2380061845 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-29T11:20:35.279 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 10 client.?.objecter ms_handle_connect 0x7fcf501dfb50 2024-03-29T11:20:35.279 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:35.279 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 10 client.?.objecter ms_handle_connect 0x7fcf5009ac40 2024-03-29T11:20:35.279 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:35.279 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 1 -- 172.21.15.196:0/2380061845 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3161335618 0 0) 0x7fcf500b7700 con 0x7fcf501dfb50 2024-03-29T11:20:35.279 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.279 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.279 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.279 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 10 monclient(hunting): my global_id is 4144 2024-03-29T11:20:35.279 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 1 -- 172.21.15.196:0/2380061845 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf3c004eb0 con 0x7fcf501dfb50 2024-03-29T11:20:35.279 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 10 client.?.objecter ms_handle_connect 0x7fcf501d7220 2024-03-29T11:20:35.279 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:35.279 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 1 -- 172.21.15.196:0/2380061845 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2256073903 0 0) 0x7fcf500b7970 con 0x7fcf5009ac40 2024-03-29T11:20:35.279 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.280 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.280 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.280 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 10 monclient(hunting): my global_id is 4139 2024-03-29T11:20:35.280 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 1 -- 172.21.15.196:0/2380061845 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf500b7700 con 0x7fcf5009ac40 2024-03-29T11:20:35.280 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 1 -- 172.21.15.196:0/2380061845 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2511612811 0 0) 0x7fcf50142eb0 con 0x7fcf501d7220 2024-03-29T11:20:35.280 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.280 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.280 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.280 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 10 monclient(hunting): my global_id is 4146 2024-03-29T11:20:35.280 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 1 -- 172.21.15.196:0/2380061845 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcf500b7970 con 0x7fcf501d7220 2024-03-29T11:20:35.280 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 1 -- 172.21.15.196:0/2380061845 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1198508217 0 0) 0x7fcf3c004eb0 con 0x7fcf501dfb50 2024-03-29T11:20:35.280 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 1 -- 172.21.15.196:0/2380061845 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf50142eb0 con 0x7fcf501dfb50 2024-03-29T11:20:35.280 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 1 -- 172.21.15.196:0/2380061845 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fcf48004130 con 0x7fcf501dfb50 2024-03-29T11:20:35.280 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.280 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:35.281 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 10 monclient(hunting): dump: 2024-03-29T11:20:35.281 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:35.281 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:35.281 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.281 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.281 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.281 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:35.281 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:35.281 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:35.281 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:35.281 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:35.281 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:35.281 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 1 -- 172.21.15.196:0/2380061845 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 724364981 0 0) 0x7fcf500b7700 con 0x7fcf5009ac40 2024-03-29T11:20:35.281 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 1 -- 172.21.15.196:0/2380061845 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf3c004eb0 con 0x7fcf5009ac40 2024-03-29T11:20:35.281 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 1 -- 172.21.15.196:0/2380061845 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fcf44001b80 con 0x7fcf5009ac40 2024-03-29T11:20:35.281 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.282 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:35.282 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 10 monclient(hunting): dump: 2024-03-29T11:20:35.282 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:35.282 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:35.282 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.282 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.282 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.282 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:35.282 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:35.282 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:35.282 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:35.282 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:35.282 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 1 -- 172.21.15.196:0/2380061845 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1075176466 0 0) 0x7fcf500b7970 con 0x7fcf501d7220 2024-03-29T11:20:35.282 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 1 -- 172.21.15.196:0/2380061845 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcf500b7700 con 0x7fcf501d7220 2024-03-29T11:20:35.283 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 1 -- 172.21.15.196:0/2380061845 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4116754194 0 0) 0x7fcf50142eb0 con 0x7fcf501dfb50 2024-03-29T11:20:35.283 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 1 -- 172.21.15.196:0/2380061845 >> v1:172.21.15.86:6790/0 conn(0x7fcf5009ac40 legacy=0x7fcf500b8100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.283 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 1 -- 172.21.15.196:0/2380061845 >> v1:172.21.15.86:6789/0 conn(0x7fcf501d7220 legacy=0x7fcf501d7640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.283 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 10 monclient: _finish_hunting 0 2024-03-29T11:20:35.283 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 1 monclient: found mon.b 2024-03-29T11:20:35.283 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:35.283 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:20:35.283 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 1 -- 172.21.15.196:0/2380061845 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf500b8810 con 0x7fcf501dfb50 2024-03-29T11:20:35.283 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 10 monclient: _finish_auth 0 2024-03-29T11:20:35.283 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 10 monclient: _check_auth_tickets 2024-03-29T11:20:35.283 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf377fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:35.283 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf557ba640 5 monclient: authenticate success, global_id 4144 2024-03-29T11:20:35.283 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf557ba640 10 monclient: _renew_subs 2024-03-29T11:20:35.283 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf557ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:20:35.283 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.286+0000 7fcf557ba640 1 -- 172.21.15.196:0/2380061845 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf500b1a30 con 0x7fcf501dfb50 2024-03-29T11:20:35.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.287+0000 7fcf557ba640 10 client.4144.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:35.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.287+0000 7fcf557ba640 10 monclient: _renew_subs 2024-03-29T11:20:35.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.287+0000 7fcf557ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:20:35.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.287+0000 7fcf557ba640 1 -- 172.21.15.196:0/2380061845 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf501d7f30 con 0x7fcf501dfb50 2024-03-29T11:20:35.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.287+0000 7fcf377fe640 1 -- 172.21.15.196:0/2380061845 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcf48004620 con 0x7fcf501dfb50 2024-03-29T11:20:35.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.287+0000 7fcf557ba640 1 librados: init done 2024-03-29T11:20:35.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.287+0000 7fcf377fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:35.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.287+0000 7fcf377fe640 1 -- 172.21.15.196:0/2380061845 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fcf48001250 con 0x7fcf501dfb50 2024-03-29T11:20:35.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.287+0000 7fcf377fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.287+0000 7fcf377fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:35.284 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.287+0000 7fcf377fe640 10 monclient: dump: 2024-03-29T11:20:35.284 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:35.284 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:35.285 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.285 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.285 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.285 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:35.285 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:35.285 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:35.285 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:35.285 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:35.285 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.287+0000 7fcf4d7fa640 4 set_mon_vals no callback set 2024-03-29T11:20:35.285 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.287+0000 7fcf377fe640 1 -- 172.21.15.196:0/2380061845 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fcf48001740 con 0x7fcf501dfb50 2024-03-29T11:20:35.285 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.287+0000 7fcf377fe640 1 -- 172.21.15.196:0/2380061845 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(7..7 src has 1..7) v4 ==== 1300+0+0 (unknown 596900893 0 0) 0x7fcf48005b40 con 0x7fcf501dfb50 2024-03-29T11:20:35.285 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.287+0000 7fcf377fe640 10 client.4144.objecter ms_dispatch 0x7fcf50000b90 osd_map(7..7 src has 1..7) v4 2024-03-29T11:20:35.285 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.287+0000 7fcf377fe640 3 client.4144.objecter handle_osd_map got epochs [7,7] > 0 2024-03-29T11:20:35.285 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.287+0000 7fcf377fe640 3 client.4144.objecter handle_osd_map decoding full epoch 7 2024-03-29T11:20:35.285 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.287+0000 7fcf377fe640 20 client.4144.objecter dump_active .. 0 homeless 2024-03-29T11:20:35.286 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.287+0000 7fcf557ba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:35.286 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.287+0000 7fcf557ba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:35.286 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.287+0000 7fcf557ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:20:35.286 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.287+0000 7fcf557ba640 1 -- 172.21.15.196:0/2380061845 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf5009b020 con 0x7fcf501dfb50 2024-03-29T11:20:35.286 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.289+0000 7fcf377fe640 1 -- 172.21.15.196:0/2380061845 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fcf501d7f30 con 0x7fcf501dfb50 2024-03-29T11:20:35.286 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.289+0000 7fcf377fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:35.286 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.289+0000 7fcf377fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:35.336 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.359+0000 7fcf557ba640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3ed0352b-96f6-4668-88dd-4747f5385eb4", "id": 5}] 2024-03-29T11:20:35.336 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.359+0000 7fcf557ba640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3ed0352b-96f6-4668-88dd-4747f5385eb4", "id": 5}] 2024-03-29T11:20:35.336 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.359+0000 7fcf557ba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:20:35.337 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.359+0000 7fcf557ba640 1 -- 172.21.15.196:0/2380061845 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd new", "uuid": "3ed0352b-96f6-4668-88dd-4747f5385eb4", "id": 5} v 0) v1 -- 0x7fcf5006ab60 con 0x7fcf501dfb50 2024-03-29T11:20:35.340 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.363+0000 7fcf377fe640 1 -- 172.21.15.196:0/2380061845 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3ed0352b-96f6-4668-88dd-4747f5385eb4", "id": 5}]=0 v8) v1 ==== 112+0+2 (unknown 3290681828 0 1988412176) 0x7fcf48001ac0 con 0x7fcf501dfb50 2024-03-29T11:20:35.340 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.363+0000 7fcf377fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3ed0352b-96f6-4668-88dd-4747f5385eb4", "id": 5}] 2024-03-29T11:20:35.340 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.363+0000 7fcf377fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:35.341 INFO:teuthology.orchestra.run.smithi196.stdout:5 2024-03-29T11:20:35.342 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.364+0000 7fcf557ba640 10 librados: watch_flush enter 2024-03-29T11:20:35.342 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.364+0000 7fcf557ba640 10 librados: watch_flush exit 2024-03-29T11:20:35.342 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.364+0000 7fcf557ba640 20 client.4144.objecter shutdown clearing up homeless session... 2024-03-29T11:20:35.342 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.364+0000 7fcf557ba640 10 client.4144.objecter successfully canceled tick 2024-03-29T11:20:35.342 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.364+0000 7fcf557ba640 10 monclient: shutdown 2024-03-29T11:20:35.342 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.364+0000 7fcf557ba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:35.342 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.364+0000 7fcf557ba640 1 -- 172.21.15.196:0/2380061845 >> v1:172.21.15.196:6789/0 conn(0x7fcf501dfb50 legacy=0x7fcf501e1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.343 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.364+0000 7fcf557ba640 1 -- 172.21.15.196:0/2380061845 shutdown_connections 2024-03-29T11:20:35.343 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.364+0000 7fcf557ba640 1 -- 172.21.15.196:0/2380061845 >> 172.21.15.196:0/2380061845 conn(0x7fcf50066b00 msgr2=0x7fcf5013c380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:35.343 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.364+0000 7fcf557ba640 1 -- 172.21.15.196:0/2380061845 shutdown_connections 2024-03-29T11:20:35.343 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.364+0000 7fcf557ba640 1 -- 172.21.15.196:0/2380061845 wait complete. 2024-03-29T11:20:35.343 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.364+0000 7fcf557ba640 1 librados: shutdown 2024-03-29T11:20:35.351 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new ae0370e8-7641-4e0b-9d5d-e2d8a6175a17 6 2024-03-29T11:20:35.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.474+0000 7f2d9d086640 10 monclient: get_monmap_and_config 2024-03-29T11:20:35.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.474+0000 7f2d9d086640 10 monclient: build_initial_monmap 2024-03-29T11:20:35.451 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.474+0000 7f2d9d086640 10 monclient: monmap: 2024-03-29T11:20:35.451 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-29T11:20:35.451 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:35.451 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:20:35.475212+0000 2024-03-29T11:20:35.451 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:20:35.475212+0000 2024-03-29T11:20:35.451 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:35.451 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:35.451 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:20:35.452 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:20:35.452 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:20:35.452 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:35.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.474+0000 7f2d9d086640 1 Processor -- start 2024-03-29T11:20:35.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.474+0000 7f2d9d086640 1 -- start start 2024-03-29T11:20:35.452 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.474+0000 7f2d9d086640 10 monclient: init 2024-03-29T11:20:35.461 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d9d086640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:35.461 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d9d086640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:35.461 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d9d086640 10 monclient(hunting): picked mon.noname-a con 0x7f2d9809ac40 addr v1:172.21.15.86:6789/0 2024-03-29T11:20:35.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d9d086640 10 monclient(hunting): picked mon.noname-c con 0x7f2d9809c870 addr v1:172.21.15.196:6789/0 2024-03-29T11:20:35.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d9d086640 10 monclient(hunting): picked mon.noname-b con 0x7f2d980b4360 addr v1:172.21.15.86:6790/0 2024-03-29T11:20:35.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d9d086640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d9809dec0 con 0x7f2d9809ac40 2024-03-29T11:20:35.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d9d086640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d9809da30 con 0x7f2d9809c870 2024-03-29T11:20:35.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d9d086640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d980b39c0 con 0x7f2d980b4360 2024-03-29T11:20:35.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d9d086640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:35.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d9d086640 10 monclient(hunting): authenticate will time out at 880.763400s 2024-03-29T11:20:35.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d9659c640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f2d9809c870 0x7f2d9813f3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:51562/0 (socket says 172.21.15.196:51562) 2024-03-29T11:20:35.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d9659c640 1 -- 172.21.15.196:0/1903398210 learned_addr learned my addr 172.21.15.196:0/1903398210 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-29T11:20:35.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d95d9b640 1 -- 172.21.15.196:0/1903398210 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3580964478 0 0) 0x7f2d9809da30 con 0x7f2d9809c870 2024-03-29T11:20:35.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d95d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d95d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d95d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d95d9b640 10 monclient(hunting): my global_id is 4147 2024-03-29T11:20:35.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d95d9b640 1 -- 172.21.15.196:0/1903398210 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d80003520 con 0x7f2d9809c870 2024-03-29T11:20:35.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d95d9b640 1 -- 172.21.15.196:0/1903398210 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 216702084 0 0) 0x7f2d980b39c0 con 0x7f2d980b4360 2024-03-29T11:20:35.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d95d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d95d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d95d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d95d9b640 10 monclient(hunting): my global_id is 4142 2024-03-29T11:20:35.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d95d9b640 1 -- 172.21.15.196:0/1903398210 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d9809da30 con 0x7f2d980b4360 2024-03-29T11:20:35.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d95d9b640 1 -- 172.21.15.196:0/1903398210 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 356927750 0 0) 0x7f2d8c001110 con 0x7f2d9809ac40 2024-03-29T11:20:35.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d95d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d95d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d95d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d95d9b640 10 monclient(hunting): my global_id is 4149 2024-03-29T11:20:35.463 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d95d9b640 1 -- 172.21.15.196:0/1903398210 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d980b39c0 con 0x7f2d9809ac40 2024-03-29T11:20:35.464 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d95d9b640 1 -- 172.21.15.196:0/1903398210 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1664768291 0 0) 0x7f2d80003520 con 0x7f2d9809c870 2024-03-29T11:20:35.464 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d95d9b640 1 -- 172.21.15.196:0/1903398210 >> v1:172.21.15.86:6790/0 conn(0x7f2d980b4360 legacy=0x7f2d9814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.464 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d95d9b640 1 -- 172.21.15.196:0/1903398210 >> v1:172.21.15.86:6789/0 conn(0x7f2d9809ac40 legacy=0x7f2d980b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.464 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d95d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:20:35.464 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d95d9b640 1 monclient: found mon.noname-c 2024-03-29T11:20:35.464 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d95d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:35.464 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d95d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T11:20:35.464 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d95d9b640 1 -- 172.21.15.196:0/1903398210 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d980b5160 con 0x7f2d9809c870 2024-03-29T11:20:35.464 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d95d9b640 10 monclient: _finish_auth 0 2024-03-29T11:20:35.464 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d95d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:20:35.464 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d95d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:35.464 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d95d9b640 1 -- 172.21.15.196:0/1903398210 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f2d88001610 con 0x7f2d9809c870 2024-03-29T11:20:35.464 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d9d086640 5 monclient: authenticate success, global_id 4147 2024-03-29T11:20:35.464 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d9d086640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:35.464 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d95d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d95d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:35.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d95d9b640 10 monclient: dump: 2024-03-29T11:20:35.465 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:35.465 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:35.465 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.465 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.465 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.465 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:35.465 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:35.465 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:35.465 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:35.465 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:35.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.475+0000 7f2d9d086640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:35.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.476+0000 7f2d95d9b640 1 -- 172.21.15.196:0/1903398210 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d88001770 con 0x7f2d9809c870 2024-03-29T11:20:35.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.476+0000 7f2d95d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:35.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.476+0000 7f2d9d086640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:35.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.476+0000 7f2d95d9b640 1 -- 172.21.15.196:0/1903398210 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f2d88001bc0 con 0x7f2d9809c870 2024-03-29T11:20:35.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.476+0000 7f2d9d086640 4 set_mon_vals no callback set 2024-03-29T11:20:35.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.476+0000 7f2d95d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.476+0000 7f2d95d9b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:35.466 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.476+0000 7f2d95d9b640 10 monclient: dump: 2024-03-29T11:20:35.466 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:35.466 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:35.466 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.466 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.466 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.466 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:35.466 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:35.466 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:35.467 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:35.467 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:35.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.476+0000 7f2d9d086640 10 monclient: shutdown 2024-03-29T11:20:35.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.476+0000 7f2d9d086640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:35.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.476+0000 7f2d9d086640 1 -- 172.21.15.196:0/1903398210 >> v1:172.21.15.196:6789/0 conn(0x7f2d9809c870 legacy=0x7f2d9813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.476+0000 7f2d9d086640 1 -- 172.21.15.196:0/1903398210 shutdown_connections 2024-03-29T11:20:35.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.476+0000 7f2d9d086640 1 -- 172.21.15.196:0/1903398210 >> 172.21.15.196:0/1903398210 conn(0x7f2d98066b00 msgr2=0x7f2d98066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:35.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.476+0000 7f2d9d086640 1 -- 172.21.15.196:0/1903398210 shutdown_connections 2024-03-29T11:20:35.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.476+0000 7f2d9d086640 1 -- 172.21.15.196:0/1903398210 wait complete. 2024-03-29T11:20:35.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.477+0000 7f2d9d086640 10 monclient: build_initial_monmap 2024-03-29T11:20:35.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.477+0000 7f2d9d086640 10 monclient: monmap: 2024-03-29T11:20:35.467 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-29T11:20:35.467 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:35.467 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 0.000000 2024-03-29T11:20:35.467 INFO:teuthology.orchestra.run.smithi196.stderr:created 0.000000 2024-03-29T11:20:35.468 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:35.468 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:35.468 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:20:35.468 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:20:35.468 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:20:35.468 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:35.468 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.477+0000 7f2d9d086640 1 librados: starting msgr at 2024-03-29T11:20:35.468 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.477+0000 7f2d9d086640 1 librados: starting objecter 2024-03-29T11:20:35.468 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.477+0000 7f2d9d086640 1 Processor -- start 2024-03-29T11:20:35.468 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.477+0000 7f2d9d086640 1 -- start start 2024-03-29T11:20:35.468 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.477+0000 7f2d9d086640 1 librados: setting wanted keys 2024-03-29T11:20:35.468 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.477+0000 7f2d9d086640 1 librados: calling monclient init 2024-03-29T11:20:35.468 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.477+0000 7f2d9d086640 10 monclient: init 2024-03-29T11:20:35.468 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d9d086640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:35.468 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d9d086640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:35.469 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d9d086640 10 monclient(hunting): picked mon.noname-a con 0x7f2d9809ac40 addr v1:172.21.15.86:6789/0 2024-03-29T11:20:35.469 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d9d086640 10 monclient(hunting): picked mon.noname-c con 0x7f2d980b11a0 addr v1:172.21.15.86:6790/0 2024-03-29T11:20:35.469 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d9d086640 10 monclient(hunting): picked mon.noname-b con 0x7f2d98139e70 addr v1:172.21.15.196:6789/0 2024-03-29T11:20:35.469 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d9d086640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d98120350 con 0x7f2d9809ac40 2024-03-29T11:20:35.469 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d9d086640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d981da720 con 0x7f2d98139e70 2024-03-29T11:20:35.469 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d9d086640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d980b1630 con 0x7f2d980b11a0 2024-03-29T11:20:35.469 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d9d086640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:35.469 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d9d086640 10 monclient(hunting): authenticate will time out at 880.766563s 2024-03-29T11:20:35.469 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d9759e640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f2d98139e70 0x7f2d9813a2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:51566/0 (socket says 172.21.15.196:51566) 2024-03-29T11:20:35.469 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d9759e640 1 -- 172.21.15.196:0/426706371 learned_addr learned my addr 172.21.15.196:0/426706371 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-29T11:20:35.469 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d7f7fe640 10 client.?.objecter ms_handle_connect 0x7f2d98139e70 2024-03-29T11:20:35.469 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d7f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:35.469 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d7f7fe640 10 client.?.objecter ms_handle_connect 0x7f2d9809ac40 2024-03-29T11:20:35.469 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d7f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:35.469 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d7f7fe640 10 client.?.objecter ms_handle_connect 0x7f2d980b11a0 2024-03-29T11:20:35.470 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d7f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:35.470 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d7f7fe640 1 -- 172.21.15.196:0/426706371 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4024981067 0 0) 0x7f2d981da720 con 0x7f2d98139e70 2024-03-29T11:20:35.470 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d7f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.470 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d7f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.470 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.470 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d7f7fe640 10 monclient(hunting): my global_id is 4150 2024-03-29T11:20:35.470 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d7f7fe640 1 -- 172.21.15.196:0/426706371 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d84004eb0 con 0x7f2d98139e70 2024-03-29T11:20:35.470 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d7f7fe640 1 -- 172.21.15.196:0/426706371 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2583944782 0 0) 0x7f2d98120350 con 0x7f2d9809ac40 2024-03-29T11:20:35.470 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d7f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.470 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d7f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.470 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.470 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d7f7fe640 10 monclient(hunting): my global_id is 4152 2024-03-29T11:20:35.470 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d7f7fe640 1 -- 172.21.15.196:0/426706371 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d981da720 con 0x7f2d9809ac40 2024-03-29T11:20:35.470 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d7f7fe640 1 -- 172.21.15.196:0/426706371 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1073100935 0 0) 0x7f2d980b1630 con 0x7f2d980b11a0 2024-03-29T11:20:35.471 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d7f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.471 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d7f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.471 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.471 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d7f7fe640 10 monclient(hunting): my global_id is 4145 2024-03-29T11:20:35.471 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.478+0000 7f2d7f7fe640 1 -- 172.21.15.196:0/426706371 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2d98120350 con 0x7f2d980b11a0 2024-03-29T11:20:35.471 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 1 -- 172.21.15.196:0/426706371 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4046576707 0 0) 0x7f2d84004eb0 con 0x7f2d98139e70 2024-03-29T11:20:35.471 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 1 -- 172.21.15.196:0/426706371 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d980b1630 con 0x7f2d98139e70 2024-03-29T11:20:35.471 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 1 -- 172.21.15.196:0/426706371 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f2d90001c50 con 0x7f2d98139e70 2024-03-29T11:20:35.471 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.471 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:35.471 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 10 monclient(hunting): dump: 2024-03-29T11:20:35.471 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:35.471 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:35.471 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.471 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.471 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.472 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:35.472 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:35.472 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:35.472 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:35.472 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:35.472 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:35.472 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 1 -- 172.21.15.196:0/426706371 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 318143557 0 0) 0x7f2d981da720 con 0x7f2d9809ac40 2024-03-29T11:20:35.472 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 1 -- 172.21.15.196:0/426706371 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d84004eb0 con 0x7f2d9809ac40 2024-03-29T11:20:35.472 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 1 -- 172.21.15.196:0/426706371 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1509069350 0 0) 0x7f2d98120350 con 0x7f2d980b11a0 2024-03-29T11:20:35.472 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 1 -- 172.21.15.196:0/426706371 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2d981da720 con 0x7f2d980b11a0 2024-03-29T11:20:35.472 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 1 -- 172.21.15.196:0/426706371 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f2d8c0018a0 con 0x7f2d9809ac40 2024-03-29T11:20:35.472 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.472 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:35.472 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 10 monclient(hunting): dump: 2024-03-29T11:20:35.472 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:35.473 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:35.473 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.473 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.473 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.473 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:35.473 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:35.473 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:35.473 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:35.473 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:35.473 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 1 -- 172.21.15.196:0/426706371 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f2d88001a80 con 0x7f2d980b11a0 2024-03-29T11:20:35.473 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.473 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:35.473 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 10 monclient(hunting): dump: 2024-03-29T11:20:35.473 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:35.474 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:35.474 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.474 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.474 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.474 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:35.474 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:35.474 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:35.474 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:35.474 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:35.474 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 1 -- 172.21.15.196:0/426706371 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 35108909 0 0) 0x7f2d980b1630 con 0x7f2d98139e70 2024-03-29T11:20:35.474 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 1 -- 172.21.15.196:0/426706371 >> v1:172.21.15.86:6790/0 conn(0x7f2d980b11a0 legacy=0x7f2d981dac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.474 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 1 -- 172.21.15.196:0/426706371 >> v1:172.21.15.86:6789/0 conn(0x7f2d9809ac40 legacy=0x7f2d980b0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.474 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:20:35.474 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 1 monclient: found mon.b 2024-03-29T11:20:35.474 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:35.475 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:20:35.475 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 1 -- 172.21.15.196:0/426706371 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d980b17c0 con 0x7f2d98139e70 2024-03-29T11:20:35.475 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:20:35.475 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:20:35.475 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:35.475 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d9d086640 5 monclient: authenticate success, global_id 4150 2024-03-29T11:20:35.475 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d9d086640 10 monclient: _renew_subs 2024-03-29T11:20:35.475 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d9d086640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:20:35.475 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d9d086640 1 -- 172.21.15.196:0/426706371 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d980ad820 con 0x7f2d98139e70 2024-03-29T11:20:35.477 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d9d086640 10 client.4150.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:35.478 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d9d086640 10 monclient: _renew_subs 2024-03-29T11:20:35.478 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d9d086640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:20:35.478 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d9d086640 1 -- 172.21.15.196:0/426706371 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d981db530 con 0x7f2d98139e70 2024-03-29T11:20:35.478 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d9d086640 1 librados: init done 2024-03-29T11:20:35.478 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 1 -- 172.21.15.196:0/426706371 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2d90003990 con 0x7f2d98139e70 2024-03-29T11:20:35.478 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:35.478 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 1 -- 172.21.15.196:0/426706371 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f2d90002100 con 0x7f2d98139e70 2024-03-29T11:20:35.478 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.478 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:35.478 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d9559a640 4 set_mon_vals no callback set 2024-03-29T11:20:35.478 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 10 monclient: dump: 2024-03-29T11:20:35.478 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:35.479 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:35.479 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.479 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.479 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.479 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:35.479 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:35.479 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:35.479 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:35.479 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:35.479 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 1 -- 172.21.15.196:0/426706371 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f2d90004170 con 0x7f2d98139e70 2024-03-29T11:20:35.479 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 1 -- 172.21.15.196:0/426706371 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(8..8 src has 1..8) v4 ==== 1415+0+0 (unknown 3906855182 0 0) 0x7f2d90006b50 con 0x7f2d98139e70 2024-03-29T11:20:35.479 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 10 client.4150.objecter ms_dispatch 0x7f2d98000b90 osd_map(8..8 src has 1..8) v4 2024-03-29T11:20:35.479 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 3 client.4150.objecter handle_osd_map got epochs [8,8] > 0 2024-03-29T11:20:35.479 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 3 client.4150.objecter handle_osd_map decoding full epoch 8 2024-03-29T11:20:35.479 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.479+0000 7f2d7f7fe640 20 client.4150.objecter dump_active .. 0 homeless 2024-03-29T11:20:35.479 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.480+0000 7f2d9d086640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:35.480 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.480+0000 7f2d9d086640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:35.480 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.480+0000 7f2d9d086640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:20:35.480 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.480+0000 7f2d9d086640 1 -- 172.21.15.196:0/426706371 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d78005080 con 0x7f2d98139e70 2024-03-29T11:20:35.480 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.482+0000 7f2d7f7fe640 1 -- 172.21.15.196:0/426706371 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f2d90004650 con 0x7f2d98139e70 2024-03-29T11:20:35.480 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.482+0000 7f2d7f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:35.480 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.482+0000 7f2d7f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:35.502 INFO:tasks.ceph.mgr.y.smithi086.stderr:2024-03-29T11:20:35.526+0000 7f4bd1e42040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:20:35.530 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.553+0000 7f2d9d086640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "ae0370e8-7641-4e0b-9d5d-e2d8a6175a17", "id": 6}] 2024-03-29T11:20:35.530 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.553+0000 7f2d9d086640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "ae0370e8-7641-4e0b-9d5d-e2d8a6175a17", "id": 6}] 2024-03-29T11:20:35.530 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.553+0000 7f2d9d086640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:20:35.530 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.553+0000 7f2d9d086640 1 -- 172.21.15.196:0/426706371 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd new", "uuid": "ae0370e8-7641-4e0b-9d5d-e2d8a6175a17", "id": 6} v 0) v1 -- 0x7f2d780058b0 con 0x7f2d98139e70 2024-03-29T11:20:35.534 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.557+0000 7f2d7f7fe640 1 -- 172.21.15.196:0/426706371 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ae0370e8-7641-4e0b-9d5d-e2d8a6175a17", "id": 6}]=0 v9) v1 ==== 112+0+2 (unknown 692276472 0 1113811849) 0x7f2d900047f0 con 0x7f2d98139e70 2024-03-29T11:20:35.534 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.557+0000 7f2d7f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "ae0370e8-7641-4e0b-9d5d-e2d8a6175a17", "id": 6}] 2024-03-29T11:20:35.534 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.557+0000 7f2d7f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:35.534 INFO:teuthology.orchestra.run.smithi196.stdout:6 2024-03-29T11:20:35.536 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.558+0000 7f2d9d086640 10 librados: watch_flush enter 2024-03-29T11:20:35.536 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.558+0000 7f2d9d086640 10 librados: watch_flush exit 2024-03-29T11:20:35.536 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.558+0000 7f2d9d086640 20 client.4150.objecter shutdown clearing up homeless session... 2024-03-29T11:20:35.536 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.558+0000 7f2d9d086640 10 client.4150.objecter successfully canceled tick 2024-03-29T11:20:35.536 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.558+0000 7f2d9d086640 10 monclient: shutdown 2024-03-29T11:20:35.536 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.558+0000 7f2d9d086640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:35.536 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.558+0000 7f2d9d086640 1 -- 172.21.15.196:0/426706371 >> v1:172.21.15.196:6789/0 conn(0x7f2d98139e70 legacy=0x7f2d9813a2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.536 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.558+0000 7f2d9d086640 1 -- 172.21.15.196:0/426706371 shutdown_connections 2024-03-29T11:20:35.536 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.558+0000 7f2d9d086640 1 -- 172.21.15.196:0/426706371 >> 172.21.15.196:0/426706371 conn(0x7f2d98066b00 msgr2=0x7f2d9813d7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:35.536 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.558+0000 7f2d9d086640 1 -- 172.21.15.196:0/426706371 shutdown_connections 2024-03-29T11:20:35.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.558+0000 7f2d9d086640 1 -- 172.21.15.196:0/426706371 wait complete. 2024-03-29T11:20:35.537 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.558+0000 7f2d9d086640 1 librados: shutdown 2024-03-29T11:20:35.546 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 3c1de7fe-970f-4063-a391-bb16258fb567 7 2024-03-29T11:20:35.562 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T11:20:35.586+0000 7fee3a432040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-29T11:20:35.637 INFO:tasks.ceph.mgr.y.smithi086.stderr:2024-03-29T11:20:35.661+0000 7f4bd1e42040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:20:35.644 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.668+0000 7f9e401a8640 10 monclient: get_monmap_and_config 2024-03-29T11:20:35.645 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.668+0000 7f9e401a8640 10 monclient: build_initial_monmap 2024-03-29T11:20:35.645 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.668+0000 7f9e401a8640 10 monclient: monmap: 2024-03-29T11:20:35.645 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-29T11:20:35.645 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:35.645 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:20:35.669329+0000 2024-03-29T11:20:35.645 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:20:35.669329+0000 2024-03-29T11:20:35.645 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:35.645 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:35.645 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:20:35.645 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:20:35.645 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:20:35.645 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:35.655 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.668+0000 7f9e401a8640 1 Processor -- start 2024-03-29T11:20:35.655 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.669+0000 7f9e401a8640 1 -- start start 2024-03-29T11:20:35.655 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.669+0000 7f9e401a8640 10 monclient: init 2024-03-29T11:20:35.655 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.669+0000 7f9e401a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:35.655 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.669+0000 7f9e401a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:35.655 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.669+0000 7f9e401a8640 10 monclient(hunting): picked mon.noname-c con 0x7f9e3809ac40 addr v1:172.21.15.196:6789/0 2024-03-29T11:20:35.655 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.669+0000 7f9e401a8640 10 monclient(hunting): picked mon.noname-b con 0x7f9e3809c870 addr v1:172.21.15.86:6790/0 2024-03-29T11:20:35.655 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.669+0000 7f9e401a8640 10 monclient(hunting): picked mon.noname-a con 0x7f9e380b4360 addr v1:172.21.15.86:6789/0 2024-03-29T11:20:35.655 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.669+0000 7f9e401a8640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e3809dec0 con 0x7f9e380b4360 2024-03-29T11:20:35.656 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.669+0000 7f9e401a8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e3809da30 con 0x7f9e3809ac40 2024-03-29T11:20:35.656 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.669+0000 7f9e401a8640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e380b39c0 con 0x7f9e3809c870 2024-03-29T11:20:35.656 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.669+0000 7f9e401a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:35.656 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.669+0000 7f9e401a8640 10 monclient(hunting): authenticate will time out at 880.957657s 2024-03-29T11:20:35.656 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.669+0000 7f9e3df44640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f9e3809ac40 0x7f9e380b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:51576/0 (socket says 172.21.15.196:51576) 2024-03-29T11:20:35.656 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.669+0000 7f9e3df44640 1 -- 172.21.15.196:0/539205451 learned_addr learned my addr 172.21.15.196:0/539205451 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-29T11:20:35.656 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 1 -- 172.21.15.196:0/539205451 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2063921889 0 0) 0x7f9e3809da30 con 0x7f9e3809ac40 2024-03-29T11:20:35.656 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.656 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.656 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.656 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 10 monclient(hunting): my global_id is 4153 2024-03-29T11:20:35.656 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 1 -- 172.21.15.196:0/539205451 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e24003520 con 0x7f9e3809ac40 2024-03-29T11:20:35.656 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 1 -- 172.21.15.196:0/539205451 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1073868251 0 0) 0x7f9e380b39c0 con 0x7f9e3809c870 2024-03-29T11:20:35.657 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.657 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.657 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.657 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 10 monclient(hunting): my global_id is 4148 2024-03-29T11:20:35.657 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 1 -- 172.21.15.196:0/539205451 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e3809da30 con 0x7f9e3809c870 2024-03-29T11:20:35.657 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 1 -- 172.21.15.196:0/539205451 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4278483590 0 0) 0x7f9e30001490 con 0x7f9e380b4360 2024-03-29T11:20:35.657 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.657 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.657 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.657 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 10 monclient(hunting): my global_id is 4155 2024-03-29T11:20:35.657 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 1 -- 172.21.15.196:0/539205451 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e380b39c0 con 0x7f9e380b4360 2024-03-29T11:20:35.657 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 1 -- 172.21.15.196:0/539205451 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1665545512 0 0) 0x7f9e24003520 con 0x7f9e3809ac40 2024-03-29T11:20:35.657 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 1 -- 172.21.15.196:0/539205451 >> v1:172.21.15.86:6790/0 conn(0x7f9e3809c870 legacy=0x7f9e3813f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.657 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 1 -- 172.21.15.196:0/539205451 >> v1:172.21.15.86:6789/0 conn(0x7f9e380b4360 legacy=0x7f9e3814d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.658 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 10 monclient: _finish_hunting 0 2024-03-29T11:20:35.658 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 1 monclient: found mon.noname-c 2024-03-29T11:20:35.658 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:35.658 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T11:20:35.658 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 1 -- 172.21.15.196:0/539205451 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e380b5160 con 0x7f9e3809ac40 2024-03-29T11:20:35.658 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 10 monclient: _finish_auth 0 2024-03-29T11:20:35.658 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 10 monclient: _check_auth_tickets 2024-03-29T11:20:35.658 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:35.658 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e401a8640 5 monclient: authenticate success, global_id 4153 2024-03-29T11:20:35.658 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 1 -- 172.21.15.196:0/539205451 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f9e34001690 con 0x7f9e3809ac40 2024-03-29T11:20:35.658 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.658 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:35.658 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 10 monclient: dump: 2024-03-29T11:20:35.658 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:35.659 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:35.659 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.659 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.659 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.659 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:35.659 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:35.659 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:35.659 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:35.659 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:35.659 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e401a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:35.660 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 1 -- 172.21.15.196:0/539205451 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e34001870 con 0x7f9e3809ac40 2024-03-29T11:20:35.660 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:35.660 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e401a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:35.660 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 1 -- 172.21.15.196:0/539205451 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f9e34001c80 con 0x7f9e3809ac40 2024-03-29T11:20:35.660 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e401a8640 4 set_mon_vals no callback set 2024-03-29T11:20:35.660 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.660 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:35.660 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e3cf42640 10 monclient: dump: 2024-03-29T11:20:35.660 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:35.660 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:35.660 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.660 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.660 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.660 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:35.661 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:35.661 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:35.661 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:35.661 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:35.661 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e401a8640 10 monclient: shutdown 2024-03-29T11:20:35.661 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e401a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:35.661 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.670+0000 7f9e401a8640 1 -- 172.21.15.196:0/539205451 >> v1:172.21.15.196:6789/0 conn(0x7f9e3809ac40 legacy=0x7f9e380b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.661 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.671+0000 7f9e401a8640 1 -- 172.21.15.196:0/539205451 shutdown_connections 2024-03-29T11:20:35.661 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.671+0000 7f9e401a8640 1 -- 172.21.15.196:0/539205451 >> 172.21.15.196:0/539205451 conn(0x7f9e38066b00 msgr2=0x7f9e38066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:35.661 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.671+0000 7f9e401a8640 1 -- 172.21.15.196:0/539205451 shutdown_connections 2024-03-29T11:20:35.661 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.671+0000 7f9e401a8640 1 -- 172.21.15.196:0/539205451 wait complete. 2024-03-29T11:20:35.662 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.671+0000 7f9e401a8640 10 monclient: build_initial_monmap 2024-03-29T11:20:35.662 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.671+0000 7f9e401a8640 10 monclient: monmap: 2024-03-29T11:20:35.662 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 0 2024-03-29T11:20:35.662 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:35.662 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 0.000000 2024-03-29T11:20:35.662 INFO:teuthology.orchestra.run.smithi196.stderr:created 0.000000 2024-03-29T11:20:35.662 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:35.662 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:35.662 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:20:35.662 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:20:35.662 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:20:35.662 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:35.662 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.671+0000 7f9e401a8640 1 librados: starting msgr at 2024-03-29T11:20:35.662 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.671+0000 7f9e401a8640 1 librados: starting objecter 2024-03-29T11:20:35.662 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.672+0000 7f9e401a8640 1 Processor -- start 2024-03-29T11:20:35.663 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.672+0000 7f9e401a8640 1 -- start start 2024-03-29T11:20:35.663 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.672+0000 7f9e401a8640 1 librados: setting wanted keys 2024-03-29T11:20:35.663 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.672+0000 7f9e401a8640 1 librados: calling monclient init 2024-03-29T11:20:35.663 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.672+0000 7f9e401a8640 10 monclient: init 2024-03-29T11:20:35.663 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.672+0000 7f9e401a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:35.663 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.672+0000 7f9e401a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:35.663 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.672+0000 7f9e401a8640 10 monclient(hunting): picked mon.noname-c con 0x7f9e3809ac40 addr v1:172.21.15.86:6790/0 2024-03-29T11:20:35.663 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.672+0000 7f9e401a8640 10 monclient(hunting): picked mon.noname-a con 0x7f9e380acf60 addr v1:172.21.15.86:6789/0 2024-03-29T11:20:35.663 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.672+0000 7f9e401a8640 10 monclient(hunting): picked mon.noname-b con 0x7f9e380b0800 addr v1:172.21.15.196:6789/0 2024-03-29T11:20:35.663 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.672+0000 7f9e401a8640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e381da7b0 con 0x7f9e380acf60 2024-03-29T11:20:35.663 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.672+0000 7f9e401a8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e380b0c70 con 0x7f9e380b0800 2024-03-29T11:20:35.663 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.672+0000 7f9e401a8640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e381db350 con 0x7f9e3809ac40 2024-03-29T11:20:35.663 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.672+0000 7f9e401a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:35.663 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.672+0000 7f9e401a8640 10 monclient(hunting): authenticate will time out at 880.960755s 2024-03-29T11:20:35.663 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.672+0000 7f9e3e745640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f9e380b0800 0x7f9e381e1b20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:51578/0 (socket says 172.21.15.196:51578) 2024-03-29T11:20:35.664 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.672+0000 7f9e3e745640 1 -- 172.21.15.196:0/1901279943 learned_addr learned my addr 172.21.15.196:0/1901279943 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-03-29T11:20:35.664 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.672+0000 7f9e2e7fc640 10 client.?.objecter ms_handle_connect 0x7f9e380b0800 2024-03-29T11:20:35.664 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.672+0000 7f9e2e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:35.664 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.672+0000 7f9e2e7fc640 10 client.?.objecter ms_handle_connect 0x7f9e3809ac40 2024-03-29T11:20:35.664 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.672+0000 7f9e2e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:35.664 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.672+0000 7f9e2e7fc640 10 client.?.objecter ms_handle_connect 0x7f9e380acf60 2024-03-29T11:20:35.664 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.672+0000 7f9e2e7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:35.664 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.672+0000 7f9e2e7fc640 1 -- 172.21.15.196:0/1901279943 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2616072871 0 0) 0x7f9e380b0c70 con 0x7f9e380b0800 2024-03-29T11:20:35.664 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.664 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.664 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.664 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 10 monclient(hunting): my global_id is 4156 2024-03-29T11:20:35.664 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 1 -- 172.21.15.196:0/1901279943 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e1c004eb0 con 0x7f9e380b0800 2024-03-29T11:20:35.664 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 1 -- 172.21.15.196:0/1901279943 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3761133271 0 0) 0x7f9e381db350 con 0x7f9e3809ac40 2024-03-29T11:20:35.665 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.665 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.665 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.665 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 10 monclient(hunting): my global_id is 4151 2024-03-29T11:20:35.665 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 1 -- 172.21.15.196:0/1901279943 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e380b0c70 con 0x7f9e3809ac40 2024-03-29T11:20:35.665 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 1 -- 172.21.15.196:0/1901279943 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3899917599 0 0) 0x7f9e381da7b0 con 0x7f9e380acf60 2024-03-29T11:20:35.665 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.665 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.665 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.665 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 10 monclient(hunting): my global_id is 4158 2024-03-29T11:20:35.665 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 1 -- 172.21.15.196:0/1901279943 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e381db350 con 0x7f9e380acf60 2024-03-29T11:20:35.665 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 1 -- 172.21.15.196:0/1901279943 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2566183661 0 0) 0x7f9e1c004eb0 con 0x7f9e380b0800 2024-03-29T11:20:35.665 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 1 -- 172.21.15.196:0/1901279943 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e381da7b0 con 0x7f9e380b0800 2024-03-29T11:20:35.665 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 1 -- 172.21.15.196:0/1901279943 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f9e30001bc0 con 0x7f9e380b0800 2024-03-29T11:20:35.666 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.666 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:35.666 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 10 monclient(hunting): dump: 2024-03-29T11:20:35.666 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:35.666 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:35.666 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.666 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.666 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.666 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:35.666 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:35.666 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:35.666 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:35.666 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:35.666 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:35.666 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 1 -- 172.21.15.196:0/1901279943 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1739170973 0 0) 0x7f9e380b0c70 con 0x7f9e3809ac40 2024-03-29T11:20:35.667 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 1 -- 172.21.15.196:0/1901279943 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e1c004eb0 con 0x7f9e3809ac40 2024-03-29T11:20:35.667 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 1 -- 172.21.15.196:0/1901279943 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f9e34001b80 con 0x7f9e3809ac40 2024-03-29T11:20:35.667 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.667 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:35.667 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 10 monclient(hunting): dump: 2024-03-29T11:20:35.667 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:35.667 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:35.667 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.667 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.667 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.667 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:35.667 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:35.667 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:35.667 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:35.667 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:35.668 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 1 -- 172.21.15.196:0/1901279943 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3338145324 0 0) 0x7f9e381db350 con 0x7f9e380acf60 2024-03-29T11:20:35.668 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 1 -- 172.21.15.196:0/1901279943 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9e380b0c70 con 0x7f9e380acf60 2024-03-29T11:20:35.668 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 1 -- 172.21.15.196:0/1901279943 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f9e28001870 con 0x7f9e380acf60 2024-03-29T11:20:35.668 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.668 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:35.668 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 10 monclient(hunting): dump: 2024-03-29T11:20:35.668 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:35.668 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:35.668 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.668 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.668 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.668 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:35.668 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:35.668 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:35.668 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:35.669 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:35.669 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 1 -- 172.21.15.196:0/1901279943 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1632706545 0 0) 0x7f9e381da7b0 con 0x7f9e380b0800 2024-03-29T11:20:35.669 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 1 -- 172.21.15.196:0/1901279943 >> v1:172.21.15.86:6790/0 conn(0x7f9e3809ac40 legacy=0x7f9e380af9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.669 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 1 -- 172.21.15.196:0/1901279943 >> v1:172.21.15.86:6789/0 conn(0x7f9e380acf60 legacy=0x7f9e381dac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.669 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:35.669 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 1 monclient: found mon.b 2024-03-29T11:20:35.669 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:35.669 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:20:35.669 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 1 -- 172.21.15.196:0/1901279943 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e381db4e0 con 0x7f9e380b0800 2024-03-29T11:20:35.669 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:35.669 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:35.669 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:35.669 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e401a8640 5 monclient: authenticate success, global_id 4156 2024-03-29T11:20:35.669 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e401a8640 10 monclient: _renew_subs 2024-03-29T11:20:35.669 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e401a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:20:35.670 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e401a8640 1 -- 172.21.15.196:0/1901279943 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e381e3240 con 0x7f9e380b0800 2024-03-29T11:20:35.670 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e401a8640 10 client.4156.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:35.670 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e401a8640 10 monclient: _renew_subs 2024-03-29T11:20:35.670 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e401a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:20:35.670 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e401a8640 1 -- 172.21.15.196:0/1901279943 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e381e35b0 con 0x7f9e380b0800 2024-03-29T11:20:35.670 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e401a8640 1 librados: init done 2024-03-29T11:20:35.670 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 1 -- 172.21.15.196:0/1901279943 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9e30002050 con 0x7f9e380b0800 2024-03-29T11:20:35.670 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:35.670 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 1 -- 172.21.15.196:0/1901279943 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f9e30003990 con 0x7f9e380b0800 2024-03-29T11:20:35.670 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.670 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:35.670 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 10 monclient: dump: 2024-03-29T11:20:35.670 INFO:teuthology.orchestra.run.smithi196.stderr:epoch 1 2024-03-29T11:20:35.670 INFO:teuthology.orchestra.run.smithi196.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:35.670 INFO:teuthology.orchestra.run.smithi196.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.671 INFO:teuthology.orchestra.run.smithi196.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.671 INFO:teuthology.orchestra.run.smithi196.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.671 INFO:teuthology.orchestra.run.smithi196.stderr:election_strategy: 1 2024-03-29T11:20:35.671 INFO:teuthology.orchestra.run.smithi196.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:35.671 INFO:teuthology.orchestra.run.smithi196.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:35.671 INFO:teuthology.orchestra.run.smithi196.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:35.671 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:20:35.671 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2ffff640 4 set_mon_vals no callback set 2024-03-29T11:20:35.671 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 1 -- 172.21.15.196:0/1901279943 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f9e30003f40 con 0x7f9e380b0800 2024-03-29T11:20:35.671 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 1 -- 172.21.15.196:0/1901279943 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(9..9 src has 1..9) v4 ==== 1530+0+0 (unknown 3509919942 0 0) 0x7f9e300068d0 con 0x7f9e380b0800 2024-03-29T11:20:35.671 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 10 client.4156.objecter ms_dispatch 0x7f9e38000b90 osd_map(9..9 src has 1..9) v4 2024-03-29T11:20:35.671 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 3 client.4156.objecter handle_osd_map got epochs [9,9] > 0 2024-03-29T11:20:35.671 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 3 client.4156.objecter handle_osd_map decoding full epoch 9 2024-03-29T11:20:35.671 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.673+0000 7f9e2e7fc640 20 client.4156.objecter dump_active .. 0 homeless 2024-03-29T11:20:35.671 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.674+0000 7f9e401a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:35.672 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.674+0000 7f9e401a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:35.672 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.674+0000 7f9e401a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:20:35.672 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.674+0000 7f9e401a8640 1 -- 172.21.15.196:0/1901279943 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e20005080 con 0x7f9e380b0800 2024-03-29T11:20:35.672 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.676+0000 7f9e2e7fc640 1 -- 172.21.15.196:0/1901279943 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f9e30005b40 con 0x7f9e380b0800 2024-03-29T11:20:35.672 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.676+0000 7f9e2e7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:35.672 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.676+0000 7f9e2e7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:35.703 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T11:20:35.726+0000 7fee3a432040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-29T11:20:35.724 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.747+0000 7f9e401a8640 10 start_mon_command cmd=[{"prefix": "osd new", "uuid": "3c1de7fe-970f-4063-a391-bb16258fb567", "id": 7}] 2024-03-29T11:20:35.724 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.747+0000 7f9e401a8640 10 monclient: _send_command 2 [{"prefix": "osd new", "uuid": "3c1de7fe-970f-4063-a391-bb16258fb567", "id": 7}] 2024-03-29T11:20:35.724 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.747+0000 7f9e401a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:20:35.724 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.747+0000 7f9e401a8640 1 -- 172.21.15.196:0/1901279943 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd new", "uuid": "3c1de7fe-970f-4063-a391-bb16258fb567", "id": 7} v 0) v1 -- 0x7f9e200058b0 con 0x7f9e380b0800 2024-03-29T11:20:35.727 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.751+0000 7f9e2e7fc640 1 -- 172.21.15.196:0/1901279943 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd new", "uuid": "3c1de7fe-970f-4063-a391-bb16258fb567", "id": 7}]=0 v10) v1 ==== 112+0+2 (unknown 665515540 0 1371665406) 0x7f9e30006b10 con 0x7f9e380b0800 2024-03-29T11:20:35.727 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.751+0000 7f9e2e7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd new", "uuid": "3c1de7fe-970f-4063-a391-bb16258fb567", "id": 7}] 2024-03-29T11:20:35.727 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.751+0000 7f9e2e7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:35.728 INFO:teuthology.orchestra.run.smithi196.stdout:7 2024-03-29T11:20:35.729 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.752+0000 7f9e401a8640 10 librados: watch_flush enter 2024-03-29T11:20:35.729 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.752+0000 7f9e401a8640 10 librados: watch_flush exit 2024-03-29T11:20:35.729 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.752+0000 7f9e401a8640 20 client.4156.objecter shutdown clearing up homeless session... 2024-03-29T11:20:35.729 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.752+0000 7f9e401a8640 10 client.4156.objecter successfully canceled tick 2024-03-29T11:20:35.729 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.752+0000 7f9e401a8640 10 monclient: shutdown 2024-03-29T11:20:35.729 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.752+0000 7f9e401a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:35.729 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.752+0000 7f9e401a8640 1 -- 172.21.15.196:0/1901279943 >> v1:172.21.15.196:6789/0 conn(0x7f9e380b0800 legacy=0x7f9e381e1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.729 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.752+0000 7f9e401a8640 1 -- 172.21.15.196:0/1901279943 shutdown_connections 2024-03-29T11:20:35.729 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.752+0000 7f9e401a8640 1 -- 172.21.15.196:0/1901279943 >> 172.21.15.196:0/1901279943 conn(0x7f9e38066b00 msgr2=0x7f9e380b4a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:35.730 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.752+0000 7f9e401a8640 1 -- 172.21.15.196:0/1901279943 shutdown_connections 2024-03-29T11:20:35.730 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.752+0000 7f9e401a8640 1 -- 172.21.15.196:0/1901279943 wait complete. 2024-03-29T11:20:35.730 INFO:teuthology.orchestra.run.smithi196.stderr:2024-03-29T11:20:35.752+0000 7f9e401a8640 1 librados: shutdown 2024-03-29T11:20:35.739 INFO:tasks.ceph.osd.0:Restarting daemon 2024-03-29T11:20:35.739 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-03-29T11:20:35.741 INFO:tasks.ceph.osd.0:Started 2024-03-29T11:20:35.741 INFO:tasks.ceph.osd.1:Restarting daemon 2024-03-29T11:20:35.742 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-03-29T11:20:35.744 INFO:tasks.ceph.osd.1:Started 2024-03-29T11:20:35.744 INFO:tasks.ceph.osd.2:Restarting daemon 2024-03-29T11:20:35.744 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-03-29T11:20:35.746 INFO:tasks.ceph.osd.2:Started 2024-03-29T11:20:35.746 INFO:tasks.ceph.osd.3:Restarting daemon 2024-03-29T11:20:35.746 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-03-29T11:20:35.748 INFO:tasks.ceph.osd.3:Started 2024-03-29T11:20:35.748 INFO:tasks.ceph.osd.4:Restarting daemon 2024-03-29T11:20:35.749 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-03-29T11:20:35.764 INFO:tasks.ceph.mgr.y.smithi086.stderr:2024-03-29T11:20:35.788+0000 7f4bd1e42040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:20:35.781 INFO:tasks.ceph.osd.4:Started 2024-03-29T11:20:35.781 INFO:tasks.ceph.osd.5:Restarting daemon 2024-03-29T11:20:35.781 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-03-29T11:20:35.783 INFO:tasks.ceph.osd.5:Started 2024-03-29T11:20:35.783 INFO:tasks.ceph.osd.6:Restarting daemon 2024-03-29T11:20:35.783 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-03-29T11:20:35.785 INFO:tasks.ceph.osd.6:Started 2024-03-29T11:20:35.785 INFO:tasks.ceph.osd.7:Restarting daemon 2024-03-29T11:20:35.786 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-03-29T11:20:35.788 INFO:tasks.ceph.osd.7:Started 2024-03-29T11:20:35.788 DEBUG:teuthology.orchestra.run.smithi086:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:35.833 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T11:20:35.856+0000 7fee3a432040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-29T11:20:35.836 INFO:tasks.ceph.mgr.y.smithi086.stderr:2024-03-29T11:20:35.860+0000 7f4bd1e42040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:20:35.885 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T11:20:35.908+0000 7fee3a432040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-29T11:20:35.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.926+0000 7f4f3607b640 10 monclient: get_monmap_and_config 2024-03-29T11:20:35.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.926+0000 7f4f3607b640 10 monclient: build_initial_monmap 2024-03-29T11:20:35.903 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.926+0000 7f4f3607b640 10 monclient: monmap: 2024-03-29T11:20:35.903 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-29T11:20:35.903 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:35.903 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T11:20:35.927603+0000 2024-03-29T11:20:35.903 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T11:20:35.927603+0000 2024-03-29T11:20:35.903 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:35.903 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:35.903 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:20:35.904 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:20:35.904 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:20:35.904 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T11:20:35.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.927+0000 7f4f3607b640 1 Processor -- start 2024-03-29T11:20:35.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.927+0000 7f4f3607b640 1 -- start start 2024-03-29T11:20:35.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.927+0000 7f4f3607b640 10 monclient: init 2024-03-29T11:20:35.912 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f3607b640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:35.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f3607b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:20:35.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f3607b640 10 monclient(hunting): picked mon.noname-a con 0x7f4f3009ac40 addr v1:172.21.15.86:6789/0 2024-03-29T11:20:35.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f3607b640 10 monclient(hunting): picked mon.noname-c con 0x7f4f3009c870 addr v1:172.21.15.196:6789/0 2024-03-29T11:20:35.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f3607b640 10 monclient(hunting): picked mon.noname-b con 0x7f4f300b4360 addr v1:172.21.15.86:6790/0 2024-03-29T11:20:35.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f3607b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f3009dec0 con 0x7f4f3009ac40 2024-03-29T11:20:35.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f3607b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f3009da30 con 0x7f4f3009c870 2024-03-29T11:20:35.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f3607b640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f300b39c0 con 0x7f4f300b4360 2024-03-29T11:20:35.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f3607b640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:35.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f3607b640 10 monclient(hunting): authenticate will time out at 860.246163s 2024-03-29T11:20:35.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2ffff640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f4f300b4360 0x7f4f3014d9e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:60420/0 (socket says 172.21.15.86:60420) 2024-03-29T11:20:35.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2f7fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f4f3009ac40 0x7f4f300b4a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:40130/0 (socket says 172.21.15.86:40130) 2024-03-29T11:20:35.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2ffff640 1 -- 172.21.15.86:0/3287513102 learned_addr learned my addr 172.21.15.86:0/3287513102 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:20:35.913 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 1 -- 172.21.15.86:0/3287513102 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2472212603 0 0) 0x7f4f300b39c0 con 0x7f4f300b4360 2024-03-29T11:20:35.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 10 monclient(hunting): my global_id is 4178 2024-03-29T11:20:35.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 1 -- 172.21.15.86:0/3287513102 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f1c003520 con 0x7f4f300b4360 2024-03-29T11:20:35.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 1 -- 172.21.15.86:0/3287513102 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 667107628 0 0) 0x7f4f20001110 con 0x7f4f3009ac40 2024-03-29T11:20:35.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 10 monclient(hunting): my global_id is 4185 2024-03-29T11:20:35.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 1 -- 172.21.15.86:0/3287513102 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f300b39c0 con 0x7f4f3009ac40 2024-03-29T11:20:35.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 1 -- 172.21.15.86:0/3287513102 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 585910909 0 0) 0x7f4f3009da30 con 0x7f4f3009c870 2024-03-29T11:20:35.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.914 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 10 monclient(hunting): my global_id is 4180 2024-03-29T11:20:35.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 1 -- 172.21.15.86:0/3287513102 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f20001110 con 0x7f4f3009c870 2024-03-29T11:20:35.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 1 -- 172.21.15.86:0/3287513102 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3716489056 0 0) 0x7f4f1c003520 con 0x7f4f300b4360 2024-03-29T11:20:35.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 1 -- 172.21.15.86:0/3287513102 >> v1:172.21.15.196:6789/0 conn(0x7f4f3009c870 legacy=0x7f4f3013f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 1 -- 172.21.15.86:0/3287513102 >> v1:172.21.15.86:6789/0 conn(0x7f4f3009ac40 legacy=0x7f4f300b4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:35.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 1 monclient: found mon.noname-b 2024-03-29T11:20:35.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:35.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:20:35.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 1 -- 172.21.15.86:0/3287513102 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f300b5160 con 0x7f4f300b4360 2024-03-29T11:20:35.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 10 monclient: _finish_auth 0 2024-03-29T11:20:35.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:35.915 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:35.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 1 -- 172.21.15.86:0/3287513102 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f4f28001a10 con 0x7f4f300b4360 2024-03-29T11:20:35.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f3607b640 5 monclient: authenticate success, global_id 4178 2024-03-29T11:20:35.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:35.916 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 10 monclient: dump: 2024-03-29T11:20:35.916 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T11:20:35.916 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:35.916 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.916 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.916 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.916 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:35.916 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:35.916 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:35.916 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:35.916 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T11:20:35.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f3607b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:35.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 1 -- 172.21.15.86:0/3287513102 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f28001bf0 con 0x7f4f300b4360 2024-03-29T11:20:35.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:35.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 1 -- 172.21.15.86:0/3287513102 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f4f28002040 con 0x7f4f300b4360 2024-03-29T11:20:35.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:35.917 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f2effd640 10 monclient: dump: 2024-03-29T11:20:35.917 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T11:20:35.917 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:35.917 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.917 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.917 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.917 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:35.917 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:35.917 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:35.918 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:35.918 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T11:20:35.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f3607b640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:35.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.928+0000 7f4f3607b640 4 set_mon_vals no callback set 2024-03-29T11:20:35.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.929+0000 7f4f3607b640 10 monclient: shutdown 2024-03-29T11:20:35.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.929+0000 7f4f3607b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:35.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.929+0000 7f4f3607b640 1 -- 172.21.15.86:0/3287513102 >> v1:172.21.15.86:6790/0 conn(0x7f4f300b4360 legacy=0x7f4f3014d9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.929+0000 7f4f3607b640 1 -- 172.21.15.86:0/3287513102 shutdown_connections 2024-03-29T11:20:35.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.929+0000 7f4f3607b640 1 -- 172.21.15.86:0/3287513102 >> 172.21.15.86:0/3287513102 conn(0x7f4f30066b00 msgr2=0x7f4f30066f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:35.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.929+0000 7f4f3607b640 1 -- 172.21.15.86:0/3287513102 shutdown_connections 2024-03-29T11:20:35.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.929+0000 7f4f3607b640 1 -- 172.21.15.86:0/3287513102 wait complete. 2024-03-29T11:20:35.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.930+0000 7f4f3607b640 10 monclient: build_initial_monmap 2024-03-29T11:20:35.918 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.930+0000 7f4f3607b640 10 monclient: monmap: 2024-03-29T11:20:35.919 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 0 2024-03-29T11:20:35.919 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:35.919 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 0.000000 2024-03-29T11:20:35.919 INFO:teuthology.orchestra.run.smithi086.stderr:created 0.000000 2024-03-29T11:20:35.919 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:35.919 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:35.919 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:20:35.919 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:20:35.919 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:20:35.919 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T11:20:35.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.930+0000 7f4f3607b640 1 librados: starting msgr at 2024-03-29T11:20:35.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.930+0000 7f4f3607b640 1 librados: starting objecter 2024-03-29T11:20:35.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.931+0000 7f4f3607b640 1 Processor -- start 2024-03-29T11:20:35.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.931+0000 7f4f3607b640 1 -- start start 2024-03-29T11:20:35.919 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.931+0000 7f4f3607b640 1 librados: setting wanted keys 2024-03-29T11:20:35.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.931+0000 7f4f3607b640 1 librados: calling monclient init 2024-03-29T11:20:35.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.931+0000 7f4f3607b640 10 monclient: init 2024-03-29T11:20:35.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.931+0000 7f4f3607b640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:35.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.931+0000 7f4f3607b640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:35.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f3607b640 10 monclient(hunting): picked mon.noname-c con 0x7f4f3009ac40 addr v1:172.21.15.86:6790/0 2024-03-29T11:20:35.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f3607b640 10 monclient(hunting): picked mon.noname-b con 0x7f4f30146040 addr v1:172.21.15.196:6789/0 2024-03-29T11:20:35.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f3607b640 10 monclient(hunting): picked mon.noname-a con 0x7f4f300acf70 addr v1:172.21.15.86:6789/0 2024-03-29T11:20:35.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f3607b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f300ae2f0 con 0x7f4f300acf70 2024-03-29T11:20:35.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f3607b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f300ae5d0 con 0x7f4f30146040 2024-03-29T11:20:35.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f3607b640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f301d4a20 con 0x7f4f3009ac40 2024-03-29T11:20:35.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f3607b640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:35.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f3607b640 10 monclient(hunting): authenticate will time out at 860.250072s 2024-03-29T11:20:35.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2f7fe640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f4f3009ac40 0x7f4f30138fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:60432/0 (socket says 172.21.15.86:60432) 2024-03-29T11:20:35.920 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2f7fe640 1 -- 172.21.15.86:0/3832345830 learned_addr learned my addr 172.21.15.86:0/3832345830 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:20:35.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 10 client.?.objecter ms_handle_connect 0x7f4f3009ac40 2024-03-29T11:20:35.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:35.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 10 client.?.objecter ms_handle_connect 0x7f4f300acf70 2024-03-29T11:20:35.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:35.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 1 -- 172.21.15.86:0/3832345830 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2623160115 0 0) 0x7f4f301d4a20 con 0x7f4f3009ac40 2024-03-29T11:20:35.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 10 monclient(hunting): my global_id is 4181 2024-03-29T11:20:35.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 1 -- 172.21.15.86:0/3832345830 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f10004eb0 con 0x7f4f3009ac40 2024-03-29T11:20:35.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 10 client.?.objecter ms_handle_connect 0x7f4f30146040 2024-03-29T11:20:35.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:35.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 1 -- 172.21.15.86:0/3832345830 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1393824916 0 0) 0x7f4f300ae2f0 con 0x7f4f300acf70 2024-03-29T11:20:35.921 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 10 monclient(hunting): my global_id is 4188 2024-03-29T11:20:35.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 1 -- 172.21.15.86:0/3832345830 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f301d4a20 con 0x7f4f300acf70 2024-03-29T11:20:35.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 1 -- 172.21.15.86:0/3832345830 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2119328016 0 0) 0x7f4f300ae5d0 con 0x7f4f30146040 2024-03-29T11:20:35.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:35.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:35.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:35.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 10 monclient(hunting): my global_id is 4183 2024-03-29T11:20:35.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 1 -- 172.21.15.86:0/3832345830 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f300ae2f0 con 0x7f4f30146040 2024-03-29T11:20:35.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 1 -- 172.21.15.86:0/3832345830 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2237978825 0 0) 0x7f4f10004eb0 con 0x7f4f3009ac40 2024-03-29T11:20:35.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 1 -- 172.21.15.86:0/3832345830 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4f300ae5d0 con 0x7f4f3009ac40 2024-03-29T11:20:35.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 1 -- 172.21.15.86:0/3832345830 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f4f20001b00 con 0x7f4f3009ac40 2024-03-29T11:20:35.922 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:35.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 10 monclient(hunting): dump: 2024-03-29T11:20:35.923 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T11:20:35.923 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:35.923 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.923 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.923 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.923 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:35.923 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:35.923 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:35.923 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:35.923 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T11:20:35.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:35.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 1 -- 172.21.15.86:0/3832345830 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1878439999 0 0) 0x7f4f300ae5d0 con 0x7f4f3009ac40 2024-03-29T11:20:35.923 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 1 -- 172.21.15.86:0/3832345830 >> v1:172.21.15.196:6789/0 conn(0x7f4f30146040 legacy=0x7f4f301d71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 1 -- 172.21.15.86:0/3832345830 >> v1:172.21.15.86:6789/0 conn(0x7f4f300acf70 legacy=0x7f4f301e2800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:20:35.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 1 monclient: found mon.c 2024-03-29T11:20:35.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:35.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:20:35.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 1 -- 172.21.15.86:0/3832345830 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f301396e0 con 0x7f4f3009ac40 2024-03-29T11:20:35.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 10 monclient: _finish_auth 0 2024-03-29T11:20:35.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:20:35.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f2cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:35.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f3607b640 5 monclient: authenticate success, global_id 4181 2024-03-29T11:20:35.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f3607b640 10 monclient: _renew_subs 2024-03-29T11:20:35.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f3607b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:20:35.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f3607b640 1 -- 172.21.15.86:0/3832345830 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f301398d0 con 0x7f4f3009ac40 2024-03-29T11:20:35.924 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f3607b640 10 client.4181.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:35.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f3607b640 10 monclient: _renew_subs 2024-03-29T11:20:35.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f3607b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:20:35.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f3607b640 1 -- 172.21.15.86:0/3832345830 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f301d7ab0 con 0x7f4f3009ac40 2024-03-29T11:20:35.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.932+0000 7f4f3607b640 1 librados: init done 2024-03-29T11:20:35.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.933+0000 7f4f2cff9640 1 -- 172.21.15.86:0/3832345830 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4f20001d40 con 0x7f4f3009ac40 2024-03-29T11:20:35.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.933+0000 7f4f2cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:35.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.933+0000 7f4f2cff9640 1 -- 172.21.15.86:0/3832345830 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f4f20004c70 con 0x7f4f3009ac40 2024-03-29T11:20:35.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.933+0000 7f4f2cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:35.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.933+0000 7f4f2cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:35.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.933+0000 7f4f2e7fc640 4 set_mon_vals no callback set 2024-03-29T11:20:35.925 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.933+0000 7f4f2cff9640 10 monclient: dump: 2024-03-29T11:20:35.925 INFO:teuthology.orchestra.run.smithi086.stderr:epoch 1 2024-03-29T11:20:35.925 INFO:teuthology.orchestra.run.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:35.925 INFO:teuthology.orchestra.run.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.925 INFO:teuthology.orchestra.run.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:35.926 INFO:teuthology.orchestra.run.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:35.926 INFO:teuthology.orchestra.run.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:35.926 INFO:teuthology.orchestra.run.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:35.926 INFO:teuthology.orchestra.run.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:35.926 INFO:teuthology.orchestra.run.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:35.926 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T11:20:35.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.933+0000 7f4f2cff9640 1 -- 172.21.15.86:0/3832345830 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7f4f200051e0 con 0x7f4f3009ac40 2024-03-29T11:20:35.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.933+0000 7f4f2cff9640 1 -- 172.21.15.86:0/3832345830 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f4f20006ae0 con 0x7f4f3009ac40 2024-03-29T11:20:35.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.933+0000 7f4f2cff9640 10 client.4181.objecter ms_dispatch 0x7f4f30000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:35.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.933+0000 7f4f2cff9640 3 client.4181.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:35.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.933+0000 7f4f2cff9640 3 client.4181.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:35.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.933+0000 7f4f2cff9640 20 client.4181.objecter dump_active .. 0 homeless 2024-03-29T11:20:35.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.933+0000 7f4f3607b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:35.926 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.933+0000 7f4f3607b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:35.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.933+0000 7f4f3607b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:20:35.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.933+0000 7f4f3607b640 1 -- 172.21.15.86:0/3832345830 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f14005080 con 0x7f4f3009ac40 2024-03-29T11:20:35.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.935+0000 7f4f2cff9640 1 -- 172.21.15.86:0/3832345830 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7f4f20006460 con 0x7f4f3009ac40 2024-03-29T11:20:35.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.935+0000 7f4f2cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:35.927 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.935+0000 7f4f2cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:35.957 INFO:tasks.ceph.mgr.y.smithi086.stderr:2024-03-29T11:20:35.981+0000 7f4bd1e42040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:20:35.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.996+0000 7f4f3607b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:35.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.996+0000 7f4f3607b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:35.972 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.996+0000 7f4f3607b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:20:35.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.996+0000 7f4f3607b640 1 -- 172.21.15.86:0/3832345830 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f14005330 con 0x7f4f3009ac40 2024-03-29T11:20:35.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.996+0000 7f4f2cff9640 1 -- 172.21.15.86:0/3832345830 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f4f200066b0 con 0x7f4f3009ac40 2024-03-29T11:20:35.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.996+0000 7f4f2cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:35.973 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.996+0000 7f4f2cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:35.973 INFO:teuthology.orchestra.run.smithi086.stdout: 2024-03-29T11:20:35.974 INFO:teuthology.orchestra.run.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:35.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.998+0000 7f4f3607b640 10 librados: watch_flush enter 2024-03-29T11:20:35.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.998+0000 7f4f3607b640 10 librados: watch_flush exit 2024-03-29T11:20:35.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.998+0000 7f4f3607b640 20 client.4181.objecter shutdown clearing up homeless session... 2024-03-29T11:20:35.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.998+0000 7f4f3607b640 10 client.4181.objecter successfully canceled tick 2024-03-29T11:20:35.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.998+0000 7f4f3607b640 10 monclient: shutdown 2024-03-29T11:20:35.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.998+0000 7f4f3607b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:35.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.998+0000 7f4f3607b640 1 -- 172.21.15.86:0/3832345830 >> v1:172.21.15.86:6790/0 conn(0x7f4f3009ac40 legacy=0x7f4f30138fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:35.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.998+0000 7f4f3607b640 1 -- 172.21.15.86:0/3832345830 shutdown_connections 2024-03-29T11:20:35.975 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.998+0000 7f4f3607b640 1 -- 172.21.15.86:0/3832345830 >> 172.21.15.86:0/3832345830 conn(0x7f4f30066b00 msgr2=0x7f4f300b7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:35.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.998+0000 7f4f3607b640 1 -- 172.21.15.86:0/3832345830 shutdown_connections 2024-03-29T11:20:35.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.998+0000 7f4f3607b640 1 -- 172.21.15.86:0/3832345830 wait complete. 2024-03-29T11:20:35.976 INFO:teuthology.orchestra.run.smithi086.stderr:2024-03-29T11:20:35.998+0000 7f4f3607b640 1 librados: shutdown 2024-03-29T11:20:35.985 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-03-29T11:20:35.985 INFO:tasks.ceph:Waiting for OSDs to come up 2024-03-29T11:20:35.997 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T11:20:36.021+0000 7fee3a432040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:20:36.056 INFO:tasks.ceph.mgr.y.smithi086.stderr:2024-03-29T11:20:36.079+0000 7f4bd1e42040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:20:36.091 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T11:20:36.115+0000 7fee3a432040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-29T11:20:36.210 INFO:tasks.ceph.mgr.y.smithi086.stderr:2024-03-29T11:20:36.234+0000 7f4bd1e42040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:20:36.248 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T11:20:36.272+0000 7fee3a432040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-29T11:20:36.252 INFO:tasks.ceph.mgr.y.smithi086.stderr:2024-03-29T11:20:36.275+0000 7f4bd1e42040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:20:36.293 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T11:20:36.314+0000 7fee3a432040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-29T11:20:36.340 INFO:tasks.ceph.osd.0.smithi086.stderr:2024-03-29T11:20:36.363+0000 7f05967bb640 -1 Falling back to public interface 2024-03-29T11:20:36.345 INFO:tasks.ceph.osd.1.smithi086.stderr:2024-03-29T11:20:36.368+0000 7fdaf8613640 -1 Falling back to public interface 2024-03-29T11:20:36.345 INFO:tasks.ceph.osd.2.smithi086.stderr:2024-03-29T11:20:36.368+0000 7fc39780c640 -1 Falling back to public interface 2024-03-29T11:20:36.345 INFO:tasks.ceph.osd.1.smithi086.stderr:2024-03-29T11:20:36.368+0000 7fdaf75c2640 -1 PosixStack listen unable to listen on v1:0.0.0.0:6804/0: (98) Address already in use 2024-03-29T11:20:36.366 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-03-29T11:20:36.389+0000 7fea2e8ce640 -1 Falling back to public interface 2024-03-29T11:20:36.381 INFO:tasks.ceph.osd.6.smithi196.stderr:2024-03-29T11:20:36.404+0000 7f75233fa640 -1 Falling back to public interface 2024-03-29T11:20:36.382 INFO:tasks.ceph.osd.5.smithi196.stderr:2024-03-29T11:20:36.405+0000 7fa59f20d640 -1 Falling back to public interface 2024-03-29T11:20:36.401 INFO:tasks.ceph.osd.4.smithi196.stderr:2024-03-29T11:20:36.424+0000 7f28da16b640 -1 Falling back to public interface 2024-03-29T11:20:36.403 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-03-29T11:20:36.426+0000 7fb409be1640 -1 Falling back to public interface 2024-03-29T11:20:36.408 INFO:tasks.ceph.mgr.y.smithi086.stderr:2024-03-29T11:20:36.432+0000 7f4bd1e42040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:20:36.447 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T11:20:36.469+0000 7fee3a432040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-29T11:20:36.790 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:36.880 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.903+0000 7fc0552a8640 10 monclient: get_monmap_and_config 2024-03-29T11:20:36.880 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.903+0000 7fc0552a8640 10 monclient: build_initial_monmap 2024-03-29T11:20:36.880 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.903+0000 7fc0552a8640 10 monclient: monmap: 2024-03-29T11:20:36.880 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:20:36.880 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:36.881 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:20:36.904664+0000 2024-03-29T11:20:36.881 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:20:36.904664+0000 2024-03-29T11:20:36.881 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:36.881 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:36.881 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:20:36.881 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:20:36.881 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:20:36.881 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:20:36.889 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.904+0000 7fc0552a8640 1 Processor -- start 2024-03-29T11:20:36.889 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.904+0000 7fc0552a8640 1 -- start start 2024-03-29T11:20:36.889 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.904+0000 7fc0552a8640 10 monclient: init 2024-03-29T11:20:36.889 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.904+0000 7fc0552a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:36.890 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.904+0000 7fc0552a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:36.890 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.904+0000 7fc0552a8640 10 monclient(hunting): picked mon.noname-c con 0x7fc0500b4710 addr v1:172.21.15.196:6789/0 2024-03-29T11:20:36.890 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc0552a8640 10 monclient(hunting): picked mon.noname-b con 0x7fc0500b3970 addr v1:172.21.15.86:6790/0 2024-03-29T11:20:36.890 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc0552a8640 10 monclient(hunting): picked mon.noname-a con 0x7fc05013f390 addr v1:172.21.15.86:6789/0 2024-03-29T11:20:36.890 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc0552a8640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0500af020 con 0x7fc05013f390 2024-03-29T11:20:36.890 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc0552a8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0500b0910 con 0x7fc0500b4710 2024-03-29T11:20:36.890 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc0552a8640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0500b0bf0 con 0x7fc0500b3970 2024-03-29T11:20:36.890 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc0552a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:36.890 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc0552a8640 10 monclient(hunting): authenticate will time out at 861.223054s 2024-03-29T11:20:36.890 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc047fff640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7fc0500b3970 0x7fc0500af4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:60438/0 (socket says 172.21.15.86:60438) 2024-03-29T11:20:36.890 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc04f7fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc05013f390 0x7fc05013f800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:40150/0 (socket says 172.21.15.86:40150) 2024-03-29T11:20:36.890 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc047fff640 1 -- 172.21.15.86:0/172473657 learned_addr learned my addr 172.21.15.86:0/172473657 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:20:36.890 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc04e7fc640 1 -- 172.21.15.86:0/172473657 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3503975494 0 0) 0x7fc0500b0bf0 con 0x7fc0500b3970 2024-03-29T11:20:36.890 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc04e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:36.891 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc04e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:36.891 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc04e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:36.891 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc04e7fc640 10 monclient(hunting): my global_id is 4184 2024-03-29T11:20:36.891 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc04e7fc640 1 -- 172.21.15.86:0/172473657 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc03c003520 con 0x7fc0500b3970 2024-03-29T11:20:36.891 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc04e7fc640 1 -- 172.21.15.86:0/172473657 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3814294763 0 0) 0x7fc0500af020 con 0x7fc05013f390 2024-03-29T11:20:36.891 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc04e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:36.891 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc04e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:36.891 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc04e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:36.891 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc04e7fc640 10 monclient(hunting): my global_id is 4191 2024-03-29T11:20:36.891 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc04e7fc640 1 -- 172.21.15.86:0/172473657 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc0500b0bf0 con 0x7fc05013f390 2024-03-29T11:20:36.891 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc04e7fc640 1 -- 172.21.15.86:0/172473657 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2391951942 0 0) 0x7fc0500b0bf0 con 0x7fc05013f390 2024-03-29T11:20:36.891 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc04e7fc640 1 -- 172.21.15.86:0/172473657 >> v1:172.21.15.86:6790/0 conn(0x7fc0500b3970 legacy=0x7fc0500af4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:36.891 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc04e7fc640 1 -- 172.21.15.86:0/172473657 >> v1:172.21.15.196:6789/0 conn(0x7fc0500b4710 legacy=0x7fc0500b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:36.891 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc04e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:36.891 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc04e7fc640 1 monclient: found mon.noname-a 2024-03-29T11:20:36.892 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc04e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:36.892 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc04e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.86:6789/0 2024-03-29T11:20:36.892 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc04e7fc640 1 -- 172.21.15.86:0/172473657 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0500aeb00 con 0x7fc05013f390 2024-03-29T11:20:36.892 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc04e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:36.892 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc04e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:36.892 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc04e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:36.892 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc0552a8640 5 monclient: authenticate success, global_id 4191 2024-03-29T11:20:36.892 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc04e7fc640 1 -- 172.21.15.86:0/172473657 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc0480019f0 con 0x7fc05013f390 2024-03-29T11:20:36.892 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc0552a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:36.892 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc04e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:36.892 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc04e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:36.892 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc04e7fc640 10 monclient: dump: 2024-03-29T11:20:36.892 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:20:36.892 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:36.892 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:36.893 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:36.893 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:36.893 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:36.893 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:36.893 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:36.893 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:36.893 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:20:36.893 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc0552a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:36.893 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc04e7fc640 1 -- 172.21.15.86:0/172473657 <== mon.0 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc048001bd0 con 0x7fc05013f390 2024-03-29T11:20:36.893 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc04e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:36.893 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc0552a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:36.893 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc04e7fc640 1 -- 172.21.15.86:0/172473657 <== mon.0 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc048002020 con 0x7fc05013f390 2024-03-29T11:20:36.893 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc0552a8640 4 set_mon_vals no callback set 2024-03-29T11:20:36.893 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc04e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:36.893 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc04e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:36.893 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.905+0000 7fc04e7fc640 10 monclient: dump: 2024-03-29T11:20:36.894 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:20:36.894 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:36.894 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:36.894 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:36.894 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:36.894 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:36.894 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:36.894 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:36.894 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:36.894 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:20:36.894 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.906+0000 7fc0552a8640 10 monclient: shutdown 2024-03-29T11:20:36.894 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.906+0000 7fc0552a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:36.894 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.906+0000 7fc0552a8640 1 -- 172.21.15.86:0/172473657 >> v1:172.21.15.86:6789/0 conn(0x7fc05013f390 legacy=0x7fc05013f800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:36.894 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.906+0000 7fc0552a8640 1 -- 172.21.15.86:0/172473657 shutdown_connections 2024-03-29T11:20:36.895 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.906+0000 7fc0552a8640 1 -- 172.21.15.86:0/172473657 >> 172.21.15.86:0/172473657 conn(0x7fc050066a00 msgr2=0x7fc050066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:36.895 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.906+0000 7fc0552a8640 1 -- 172.21.15.86:0/172473657 shutdown_connections 2024-03-29T11:20:36.895 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.906+0000 7fc0552a8640 1 -- 172.21.15.86:0/172473657 wait complete. 2024-03-29T11:20:36.895 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.906+0000 7fc0552a8640 10 monclient: build_initial_monmap 2024-03-29T11:20:36.895 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.906+0000 7fc0552a8640 10 monclient: monmap: 2024-03-29T11:20:36.895 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:20:36.895 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:36.895 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:20:36.895 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:20:36.895 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:36.895 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:36.895 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:20:36.895 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:20:36.895 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:20:36.895 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:20:36.895 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.907+0000 7fc0552a8640 1 librados: starting msgr at 2024-03-29T11:20:36.896 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.907+0000 7fc0552a8640 1 librados: starting objecter 2024-03-29T11:20:36.896 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.907+0000 7fc0552a8640 1 Processor -- start 2024-03-29T11:20:36.896 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.907+0000 7fc0552a8640 1 -- start start 2024-03-29T11:20:36.896 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.907+0000 7fc0552a8640 1 librados: setting wanted keys 2024-03-29T11:20:36.896 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.907+0000 7fc0552a8640 1 librados: calling monclient init 2024-03-29T11:20:36.896 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.907+0000 7fc0552a8640 10 monclient: init 2024-03-29T11:20:36.896 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.907+0000 7fc0552a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:36.896 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.907+0000 7fc0552a8640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:20:36.896 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.907+0000 7fc0552a8640 10 monclient(hunting): picked mon.noname-a con 0x7fc0500b3970 addr v1:172.21.15.86:6789/0 2024-03-29T11:20:36.896 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.907+0000 7fc0552a8640 10 monclient(hunting): picked mon.noname-c con 0x7fc0500b4710 addr v1:172.21.15.86:6790/0 2024-03-29T11:20:36.896 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.907+0000 7fc0552a8640 10 monclient(hunting): picked mon.noname-b con 0x7fc05013f390 addr v1:172.21.15.196:6789/0 2024-03-29T11:20:36.896 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.907+0000 7fc0552a8640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc050134cf0 con 0x7fc0500b3970 2024-03-29T11:20:36.896 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.907+0000 7fc0552a8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc050136ba0 con 0x7fc05013f390 2024-03-29T11:20:36.896 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.907+0000 7fc0552a8640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc050135d90 con 0x7fc0500b4710 2024-03-29T11:20:36.897 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.907+0000 7fc0552a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:36.897 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.907+0000 7fc0552a8640 10 monclient(hunting): authenticate will time out at 861.225987s 2024-03-29T11:20:36.897 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.907+0000 7fc047fff640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7fc0500b4710 0x7fc050136490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:60448/0 (socket says 172.21.15.86:60448) 2024-03-29T11:20:36.897 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc047fff640 1 -- 172.21.15.86:0/941868435 learned_addr learned my addr 172.21.15.86:0/941868435 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:20:36.897 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc04effd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc0500b3970 0x7fc050142af0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:40158/0 (socket says 172.21.15.86:40158) 2024-03-29T11:20:36.897 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 client.?.objecter ms_handle_connect 0x7fc0500b4710 2024-03-29T11:20:36.897 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:36.897 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 client.?.objecter ms_handle_connect 0x7fc0500b3970 2024-03-29T11:20:36.897 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:36.897 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 1 -- 172.21.15.86:0/941868435 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4211574992 0 0) 0x7fc050135d90 con 0x7fc0500b4710 2024-03-29T11:20:36.897 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:36.897 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:36.897 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:36.897 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 monclient(hunting): my global_id is 4187 2024-03-29T11:20:36.897 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 1 -- 172.21.15.86:0/941868435 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc030004eb0 con 0x7fc0500b4710 2024-03-29T11:20:36.898 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 client.?.objecter ms_handle_connect 0x7fc05013f390 2024-03-29T11:20:36.898 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:36.898 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 1 -- 172.21.15.86:0/941868435 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 652199851 0 0) 0x7fc050134cf0 con 0x7fc0500b3970 2024-03-29T11:20:36.898 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:36.898 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:36.898 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:36.898 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 monclient(hunting): my global_id is 4194 2024-03-29T11:20:36.898 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 1 -- 172.21.15.86:0/941868435 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc050135d90 con 0x7fc0500b3970 2024-03-29T11:20:36.898 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 1 -- 172.21.15.86:0/941868435 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1282090993 0 0) 0x7fc050136ba0 con 0x7fc05013f390 2024-03-29T11:20:36.898 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:36.898 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:36.898 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:36.898 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 monclient(hunting): my global_id is 4189 2024-03-29T11:20:36.898 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 1 -- 172.21.15.86:0/941868435 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc050134cf0 con 0x7fc05013f390 2024-03-29T11:20:36.899 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 1 -- 172.21.15.86:0/941868435 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3815504019 0 0) 0x7fc050135d90 con 0x7fc0500b3970 2024-03-29T11:20:36.899 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 1 -- 172.21.15.86:0/941868435 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc050136ba0 con 0x7fc0500b3970 2024-03-29T11:20:36.899 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 1 -- 172.21.15.86:0/941868435 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2225972486 0 0) 0x7fc030004eb0 con 0x7fc0500b4710 2024-03-29T11:20:36.899 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 1 -- 172.21.15.86:0/941868435 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc050135d90 con 0x7fc0500b4710 2024-03-29T11:20:36.899 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 1 -- 172.21.15.86:0/941868435 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc040001870 con 0x7fc0500b3970 2024-03-29T11:20:36.899 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:36.899 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:36.899 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 monclient(hunting): dump: 2024-03-29T11:20:36.899 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:20:36.899 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:36.899 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:36.899 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:36.899 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:36.899 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:36.900 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:36.900 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:36.900 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:36.900 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:20:36.900 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:36.900 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 1 -- 172.21.15.86:0/941868435 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc0380019a0 con 0x7fc0500b4710 2024-03-29T11:20:36.900 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:36.900 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:36.900 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 monclient(hunting): dump: 2024-03-29T11:20:36.900 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:20:36.900 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:36.900 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:36.900 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:36.900 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:36.900 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:36.901 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:36.901 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:36.901 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:36.901 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:20:36.901 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 1 -- 172.21.15.86:0/941868435 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2414539804 0 0) 0x7fc050134cf0 con 0x7fc05013f390 2024-03-29T11:20:36.901 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 1 -- 172.21.15.86:0/941868435 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc030004eb0 con 0x7fc05013f390 2024-03-29T11:20:36.901 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 1 -- 172.21.15.86:0/941868435 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc048001ee0 con 0x7fc05013f390 2024-03-29T11:20:36.901 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:36.901 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:36.901 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 monclient(hunting): dump: 2024-03-29T11:20:36.901 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:20:36.901 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:36.901 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:36.901 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:36.901 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:36.902 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:36.902 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:36.902 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:36.902 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:36.902 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:20:36.902 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 1 -- 172.21.15.86:0/941868435 <== mon.0 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1628842751 0 0) 0x7fc050136ba0 con 0x7fc0500b3970 2024-03-29T11:20:36.902 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 1 -- 172.21.15.86:0/941868435 >> v1:172.21.15.86:6790/0 conn(0x7fc0500b4710 legacy=0x7fc050136490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:36.902 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 1 -- 172.21.15.86:0/941868435 >> v1:172.21.15.196:6789/0 conn(0x7fc05013f390 legacy=0x7fc0501e2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:36.902 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 monclient: _finish_hunting 0 2024-03-29T11:20:36.902 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 1 monclient: found mon.a 2024-03-29T11:20:36.902 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:36.902 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:20:36.902 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 1 -- 172.21.15.86:0/941868435 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc050136070 con 0x7fc0500b3970 2024-03-29T11:20:36.902 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 monclient: _finish_auth 0 2024-03-29T11:20:36.902 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 monclient: _check_auth_tickets 2024-03-29T11:20:36.903 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:36.903 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0552a8640 5 monclient: authenticate success, global_id 4194 2024-03-29T11:20:36.903 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0552a8640 10 monclient: _renew_subs 2024-03-29T11:20:36.903 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0552a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:20:36.903 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0552a8640 1 -- 172.21.15.86:0/941868435 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc050134fd0 con 0x7fc0500b3970 2024-03-29T11:20:36.905 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0552a8640 10 client.4194.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:36.905 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0552a8640 10 monclient: _renew_subs 2024-03-29T11:20:36.905 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 1 -- 172.21.15.86:0/941868435 <== mon.0 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc040003610 con 0x7fc0500b3970 2024-03-29T11:20:36.905 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0552a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:20:36.905 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0552a8640 1 -- 172.21.15.86:0/941868435 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc050141260 con 0x7fc0500b3970 2024-03-29T11:20:36.905 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0552a8640 1 librados: init done 2024-03-29T11:20:36.905 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:36.906 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 1 -- 172.21.15.86:0/941868435 <== mon.0 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc0400039a0 con 0x7fc0500b3970 2024-03-29T11:20:36.906 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:36.906 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:36.906 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 monclient: dump: 2024-03-29T11:20:36.906 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:20:36.906 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:36.906 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:36.906 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:36.906 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:36.906 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:36.906 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:36.906 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:36.906 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:36.906 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:20:36.907 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc04dffb640 4 set_mon_vals no callback set 2024-03-29T11:20:36.907 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 1 -- 172.21.15.86:0/941868435 <== mon.0 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 1) v1 ==== 807+0+0 (unknown 3189685207 0 0) 0x7fc040003f10 con 0x7fc0500b3970 2024-03-29T11:20:36.907 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 1 -- 172.21.15.86:0/941868435 <== mon.0 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7fc0400041c0 con 0x7fc0500b3970 2024-03-29T11:20:36.907 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 10 client.4194.objecter ms_dispatch 0x7fc050000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:36.907 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 3 client.4194.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:36.907 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.908+0000 7fc0477fe640 3 client.4194.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:36.907 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.909+0000 7fc0477fe640 20 client.4194.objecter dump_active .. 0 homeless 2024-03-29T11:20:36.907 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.909+0000 7fc0552a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:36.907 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.909+0000 7fc0552a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:36.907 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.909+0000 7fc0552a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:20:36.907 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.909+0000 7fc0552a8640 1 -- 172.21.15.86:0/941868435 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0500688a0 con 0x7fc0500b3970 2024-03-29T11:20:36.907 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.911+0000 7fc0477fe640 1 -- 172.21.15.86:0/941868435 <== mon.0 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (unknown 1092875540 0 2031569661) 0x7fc0400043d0 con 0x7fc0500b3970 2024-03-29T11:20:36.907 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.911+0000 7fc0477fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:36.907 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.911+0000 7fc0477fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:36.949 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.972+0000 7fc0552a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:36.949 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.972+0000 7fc0552a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:36.949 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.972+0000 7fc0552a8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:20:36.949 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.972+0000 7fc0552a8640 1 -- 172.21.15.86:0/941868435 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc050075f10 con 0x7fc0500b3970 2024-03-29T11:20:36.949 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.972+0000 7fc0477fe640 1 -- 172.21.15.86:0/941868435 <== mon.0 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7fc0400057a0 con 0x7fc0500b3970 2024-03-29T11:20:36.949 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.972+0000 7fc0477fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:36.949 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.972+0000 7fc0477fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:36.949 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:20:36.949 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:36.951 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.973+0000 7fc0552a8640 10 librados: watch_flush enter 2024-03-29T11:20:36.951 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.973+0000 7fc0552a8640 10 librados: watch_flush exit 2024-03-29T11:20:36.951 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.973+0000 7fc0552a8640 20 client.4194.objecter shutdown clearing up homeless session... 2024-03-29T11:20:36.951 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.973+0000 7fc0552a8640 10 client.4194.objecter successfully canceled tick 2024-03-29T11:20:36.951 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.973+0000 7fc0552a8640 10 monclient: shutdown 2024-03-29T11:20:36.951 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.973+0000 7fc0552a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:36.951 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.973+0000 7fc0552a8640 1 -- 172.21.15.86:0/941868435 >> v1:172.21.15.86:6789/0 conn(0x7fc0500b3970 legacy=0x7fc050142af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:36.951 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.973+0000 7fc0552a8640 1 -- 172.21.15.86:0/941868435 shutdown_connections 2024-03-29T11:20:36.951 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.973+0000 7fc0552a8640 1 -- 172.21.15.86:0/941868435 >> 172.21.15.86:0/941868435 conn(0x7fc050066a00 msgr2=0x7fc0500b6190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:36.951 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.973+0000 7fc0552a8640 1 -- 172.21.15.86:0/941868435 shutdown_connections 2024-03-29T11:20:36.951 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.973+0000 7fc0552a8640 1 -- 172.21.15.86:0/941868435 wait complete. 2024-03-29T11:20:36.951 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:36.973+0000 7fc0552a8640 1 librados: shutdown 2024-03-29T11:20:36.958 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:37.507 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T11:20:37.531+0000 7fee3a432040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:20:37.518 INFO:tasks.ceph.mgr.y.smithi086.stderr:2024-03-29T11:20:37.539+0000 7f4bd1e42040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-29T11:20:37.780 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T11:20:37.803+0000 7fee3a432040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:20:37.790 INFO:tasks.ceph.mgr.y.smithi086.stderr:2024-03-29T11:20:37.814+0000 7f4bd1e42040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-29T11:20:37.831 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T11:20:37.855+0000 7fee3a432040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:20:37.841 INFO:tasks.ceph.mgr.y.smithi086.stderr:2024-03-29T11:20:37.864+0000 7f4bd1e42040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-29T11:20:37.945 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T11:20:37.968+0000 7fee3a432040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:20:37.954 INFO:tasks.ceph.mgr.y.smithi086.stderr:2024-03-29T11:20:37.978+0000 7f4bd1e42040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-29T11:20:38.966 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T11:20:38.989+0000 7fedda9fc640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-03-29T11:20:38.966 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-03-29T11:20:38.989+0000 7fedda9fc640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-03-29T11:20:39.855 INFO:tasks.ceph.osd.6.smithi196.stderr:2024-03-29T11:20:39.879+0000 7f75233fa640 -1 osd.6 0 log_to_monitors true 2024-03-29T11:20:39.861 INFO:tasks.ceph.osd.2.smithi086.stderr:2024-03-29T11:20:39.884+0000 7fc39780c640 -1 osd.2 0 log_to_monitors true 2024-03-29T11:20:39.866 INFO:tasks.ceph.osd.0.smithi086.stderr:2024-03-29T11:20:39.890+0000 7f05967bb640 -1 osd.0 0 log_to_monitors true 2024-03-29T11:20:39.867 INFO:tasks.ceph.osd.1.smithi086.stderr:2024-03-29T11:20:39.890+0000 7fdaf8613640 -1 osd.1 0 log_to_monitors true 2024-03-29T11:20:39.885 INFO:tasks.ceph.osd.5.smithi196.stderr:2024-03-29T11:20:39.908+0000 7fa59f20d640 -1 osd.5 0 log_to_monitors true 2024-03-29T11:20:39.906 INFO:tasks.ceph.osd.3.smithi086.stderr:2024-03-29T11:20:39.930+0000 7fea2e8ce640 -1 osd.3 0 log_to_monitors true 2024-03-29T11:20:39.918 INFO:tasks.ceph.osd.4.smithi196.stderr:2024-03-29T11:20:39.942+0000 7f28da16b640 -1 osd.4 0 log_to_monitors true 2024-03-29T11:20:39.940 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-03-29T11:20:39.963+0000 7fb409be1640 -1 osd.7 0 log_to_monitors true 2024-03-29T11:20:43.766 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:43.914 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.938+0000 7fc5cffdb640 10 monclient: get_monmap_and_config 2024-03-29T11:20:43.915 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.938+0000 7fc5cffdb640 10 monclient: build_initial_monmap 2024-03-29T11:20:43.915 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.938+0000 7fc5cffdb640 10 monclient: monmap: 2024-03-29T11:20:43.915 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:20:43.915 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:43.915 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:20:43.939021+0000 2024-03-29T11:20:43.915 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:20:43.939021+0000 2024-03-29T11:20:43.915 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:43.915 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:43.915 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:20:43.915 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:20:43.915 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:20:43.915 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:20:43.916 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.939+0000 7fc5cffdb640 1 Processor -- start 2024-03-29T11:20:43.922 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.939+0000 7fc5cffdb640 1 -- start start 2024-03-29T11:20:43.922 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.939+0000 7fc5cffdb640 10 monclient: init 2024-03-29T11:20:43.922 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.941+0000 7fc5cffdb640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:43.922 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.941+0000 7fc5cffdb640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:43.922 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.941+0000 7fc5cffdb640 10 monclient(hunting): picked mon.noname-b con 0x7fc5c81457f0 addr v1:172.21.15.86:6790/0 2024-03-29T11:20:43.922 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.941+0000 7fc5cffdb640 10 monclient(hunting): picked mon.noname-c con 0x7fc5c813f650 addr v1:172.21.15.196:6789/0 2024-03-29T11:20:43.922 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.941+0000 7fc5cffdb640 10 monclient(hunting): picked mon.noname-a con 0x7fc5c80a9d70 addr v1:172.21.15.86:6789/0 2024-03-29T11:20:43.922 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.941+0000 7fc5cffdb640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c80aaef0 con 0x7fc5c80a9d70 2024-03-29T11:20:43.923 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.941+0000 7fc5cffdb640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c8145bd0 con 0x7fc5c813f650 2024-03-29T11:20:43.923 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.941+0000 7fc5cffdb640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c8145eb0 con 0x7fc5c81457f0 2024-03-29T11:20:43.923 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.941+0000 7fc5cffdb640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:43.923 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.941+0000 7fc5cffdb640 10 monclient(hunting): authenticate will time out at 868.259526s 2024-03-29T11:20:43.923 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.941+0000 7fc5ce578640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc5c80a9d70 0x7fc5c814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:53778/0 (socket says 172.21.15.86:53778) 2024-03-29T11:20:43.923 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.941+0000 7fc5ce578640 1 -- 172.21.15.86:0/1360938448 learned_addr learned my addr 172.21.15.86:0/1360938448 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:20:43.923 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.942+0000 7fc5ccd75640 1 -- 172.21.15.86:0/1360938448 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2897977656 0 0) 0x7fc5c8145bd0 con 0x7fc5c813f650 2024-03-29T11:20:43.923 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.942+0000 7fc5ccd75640 10 monclient(hunting): _negotiate 2024-03-29T11:20:43.923 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.942+0000 7fc5ccd75640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:43.923 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.942+0000 7fc5ccd75640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:43.923 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.942+0000 7fc5ccd75640 10 monclient(hunting): my global_id is 4231 2024-03-29T11:20:43.923 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.942+0000 7fc5ccd75640 1 -- 172.21.15.86:0/1360938448 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5b4003520 con 0x7fc5c813f650 2024-03-29T11:20:43.923 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.942+0000 7fc5ccd75640 1 -- 172.21.15.86:0/1360938448 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1149194540 0 0) 0x7fc5c8145eb0 con 0x7fc5c81457f0 2024-03-29T11:20:43.923 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.942+0000 7fc5ccd75640 10 monclient(hunting): _negotiate 2024-03-29T11:20:43.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.942+0000 7fc5ccd75640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:43.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.942+0000 7fc5ccd75640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:43.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.942+0000 7fc5ccd75640 10 monclient(hunting): my global_id is 4229 2024-03-29T11:20:43.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.942+0000 7fc5ccd75640 1 -- 172.21.15.86:0/1360938448 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5c8145bd0 con 0x7fc5c81457f0 2024-03-29T11:20:43.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.943+0000 7fc5ccd75640 1 -- 172.21.15.86:0/1360938448 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1663999785 0 0) 0x7fc5c80aaef0 con 0x7fc5c80a9d70 2024-03-29T11:20:43.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.943+0000 7fc5ccd75640 10 monclient(hunting): _negotiate 2024-03-29T11:20:43.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.943+0000 7fc5ccd75640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:43.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.943+0000 7fc5ccd75640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:43.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.943+0000 7fc5ccd75640 10 monclient(hunting): my global_id is 4239 2024-03-29T11:20:43.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.943+0000 7fc5ccd75640 1 -- 172.21.15.86:0/1360938448 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5c8145eb0 con 0x7fc5c80a9d70 2024-03-29T11:20:43.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.943+0000 7fc5ccd75640 1 -- 172.21.15.86:0/1360938448 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3192859077 0 0) 0x7fc5b4003520 con 0x7fc5c813f650 2024-03-29T11:20:43.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.943+0000 7fc5ccd75640 1 -- 172.21.15.86:0/1360938448 >> v1:172.21.15.86:6790/0 conn(0x7fc5c81457f0 legacy=0x7fc5c813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:43.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.943+0000 7fc5ccd75640 1 -- 172.21.15.86:0/1360938448 >> v1:172.21.15.86:6789/0 conn(0x7fc5c80a9d70 legacy=0x7fc5c814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:43.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.943+0000 7fc5ccd75640 10 monclient: _finish_hunting 0 2024-03-29T11:20:43.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.943+0000 7fc5ccd75640 1 monclient: found mon.noname-c 2024-03-29T11:20:43.925 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.943+0000 7fc5ccd75640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:43.925 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.943+0000 7fc5ccd75640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T11:20:43.925 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.943+0000 7fc5ccd75640 1 -- 172.21.15.86:0/1360938448 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c8146340 con 0x7fc5c813f650 2024-03-29T11:20:43.925 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.943+0000 7fc5ccd75640 10 monclient: _finish_auth 0 2024-03-29T11:20:43.925 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.943+0000 7fc5ccd75640 10 monclient: _check_auth_tickets 2024-03-29T11:20:43.925 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.943+0000 7fc5ccd75640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:43.925 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.943+0000 7fc5cffdb640 5 monclient: authenticate success, global_id 4231 2024-03-29T11:20:43.925 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.943+0000 7fc5cffdb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:43.925 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.943+0000 7fc5ccd75640 1 -- 172.21.15.86:0/1360938448 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc5b8001690 con 0x7fc5c813f650 2024-03-29T11:20:43.925 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.943+0000 7fc5ccd75640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:43.925 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.943+0000 7fc5ccd75640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:43.925 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.943+0000 7fc5ccd75640 10 monclient: dump: 2024-03-29T11:20:43.925 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:20:43.925 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:43.926 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:43.926 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:43.926 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:43.926 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:43.926 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:43.926 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:43.926 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:43.926 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:20:43.926 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.944+0000 7fc5cffdb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:43.926 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.944+0000 7fc5ccd75640 1 -- 172.21.15.86:0/1360938448 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5b80018b0 con 0x7fc5c813f650 2024-03-29T11:20:43.926 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.944+0000 7fc5ccd75640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:43.926 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.944+0000 7fc5cffdb640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:43.926 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.944+0000 7fc5cffdb640 4 set_mon_vals no callback set 2024-03-29T11:20:43.926 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.944+0000 7fc5ccd75640 1 -- 172.21.15.86:0/1360938448 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc5b8001c40 con 0x7fc5c813f650 2024-03-29T11:20:43.926 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.944+0000 7fc5ccd75640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:43.927 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.944+0000 7fc5ccd75640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:43.927 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.944+0000 7fc5ccd75640 10 monclient: dump: 2024-03-29T11:20:43.927 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:20:43.927 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:43.927 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:43.927 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:43.927 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:43.927 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:43.927 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:43.927 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:43.927 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:43.927 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:20:43.927 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.944+0000 7fc5cffdb640 10 monclient: shutdown 2024-03-29T11:20:43.927 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.944+0000 7fc5cffdb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:43.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.944+0000 7fc5cffdb640 1 -- 172.21.15.86:0/1360938448 >> v1:172.21.15.196:6789/0 conn(0x7fc5c813f650 legacy=0x7fc5c80b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:43.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.945+0000 7fc5cffdb640 1 -- 172.21.15.86:0/1360938448 shutdown_connections 2024-03-29T11:20:43.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.945+0000 7fc5cffdb640 1 -- 172.21.15.86:0/1360938448 >> 172.21.15.86:0/1360938448 conn(0x7fc5c8066a00 msgr2=0x7fc5c8066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:43.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.945+0000 7fc5cffdb640 1 -- 172.21.15.86:0/1360938448 shutdown_connections 2024-03-29T11:20:43.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.945+0000 7fc5cffdb640 1 -- 172.21.15.86:0/1360938448 wait complete. 2024-03-29T11:20:43.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.946+0000 7fc5cffdb640 10 monclient: build_initial_monmap 2024-03-29T11:20:43.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.946+0000 7fc5cffdb640 10 monclient: monmap: 2024-03-29T11:20:43.928 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:20:43.928 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:43.928 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:20:43.928 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:20:43.928 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:43.928 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:43.928 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:20:43.928 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:20:43.929 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:20:43.929 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:20:43.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.946+0000 7fc5cffdb640 1 librados: starting msgr at 2024-03-29T11:20:43.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.946+0000 7fc5cffdb640 1 librados: starting objecter 2024-03-29T11:20:43.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.946+0000 7fc5cffdb640 1 Processor -- start 2024-03-29T11:20:43.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.946+0000 7fc5cffdb640 1 -- start start 2024-03-29T11:20:43.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.946+0000 7fc5cffdb640 1 librados: setting wanted keys 2024-03-29T11:20:43.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.946+0000 7fc5cffdb640 1 librados: calling monclient init 2024-03-29T11:20:43.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.946+0000 7fc5cffdb640 10 monclient: init 2024-03-29T11:20:43.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.947+0000 7fc5cffdb640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:43.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.947+0000 7fc5cffdb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:20:43.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.947+0000 7fc5cffdb640 10 monclient(hunting): picked mon.noname-b con 0x7fc5c80a9d70 addr v1:172.21.15.196:6789/0 2024-03-29T11:20:43.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.947+0000 7fc5cffdb640 10 monclient(hunting): picked mon.noname-c con 0x7fc5c813f650 addr v1:172.21.15.86:6790/0 2024-03-29T11:20:43.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.947+0000 7fc5cffdb640 10 monclient(hunting): picked mon.noname-a con 0x7fc5c81457f0 addr v1:172.21.15.86:6789/0 2024-03-29T11:20:43.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.947+0000 7fc5cffdb640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c80ace20 con 0x7fc5c81457f0 2024-03-29T11:20:43.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.947+0000 7fc5cffdb640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c81d7080 con 0x7fc5c80a9d70 2024-03-29T11:20:43.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.947+0000 7fc5cffdb640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c81d7f00 con 0x7fc5c813f650 2024-03-29T11:20:43.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.947+0000 7fc5cffdb640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:43.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.947+0000 7fc5cffdb640 10 monclient(hunting): authenticate will time out at 868.265334s 2024-03-29T11:20:43.935 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.947+0000 7fc5ce578640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc5c81457f0 0x7fc5c8137cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:53886/0 (socket says 172.21.15.86:53886) 2024-03-29T11:20:43.935 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.947+0000 7fc5ce578640 1 -- 172.21.15.86:0/2527566637 learned_addr learned my addr 172.21.15.86:0/2527566637 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:20:43.935 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.947+0000 7fc5be7fc640 10 client.?.objecter ms_handle_connect 0x7fc5c80a9d70 2024-03-29T11:20:43.935 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.947+0000 7fc5be7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:43.935 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.947+0000 7fc5be7fc640 10 client.?.objecter ms_handle_connect 0x7fc5c813f650 2024-03-29T11:20:43.935 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.947+0000 7fc5be7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:43.935 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.948+0000 7fc5be7fc640 1 -- 172.21.15.86:0/2527566637 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3364653080 0 0) 0x7fc5c81d7080 con 0x7fc5c80a9d70 2024-03-29T11:20:43.935 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.948+0000 7fc5be7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:43.935 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.948+0000 7fc5be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:43.935 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.948+0000 7fc5be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:43.935 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.948+0000 7fc5be7fc640 10 monclient(hunting): my global_id is 4234 2024-03-29T11:20:43.935 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.948+0000 7fc5be7fc640 1 -- 172.21.15.86:0/2527566637 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5ac004eb0 con 0x7fc5c80a9d70 2024-03-29T11:20:43.935 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.948+0000 7fc5be7fc640 10 client.?.objecter ms_handle_connect 0x7fc5c81457f0 2024-03-29T11:20:43.935 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.948+0000 7fc5be7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:43.936 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.948+0000 7fc5be7fc640 1 -- 172.21.15.86:0/2527566637 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1808428905 0 0) 0x7fc5ac004eb0 con 0x7fc5c80a9d70 2024-03-29T11:20:43.936 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.948+0000 7fc5be7fc640 1 -- 172.21.15.86:0/2527566637 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc5c81d7080 con 0x7fc5c80a9d70 2024-03-29T11:20:43.936 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.948+0000 7fc5be7fc640 1 -- 172.21.15.86:0/2527566637 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc5c4001910 con 0x7fc5c80a9d70 2024-03-29T11:20:43.936 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.948+0000 7fc5be7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:43.936 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.948+0000 7fc5be7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:43.936 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.948+0000 7fc5be7fc640 10 monclient(hunting): dump: 2024-03-29T11:20:43.936 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:20:43.936 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:43.936 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:43.936 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:43.936 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:43.936 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:43.936 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:43.936 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:43.936 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:43.937 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:20:43.937 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.948+0000 7fc5be7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:43.937 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.948+0000 7fc5be7fc640 1 -- 172.21.15.86:0/2527566637 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1728508628 0 0) 0x7fc5c81d7f00 con 0x7fc5c813f650 2024-03-29T11:20:43.937 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.948+0000 7fc5be7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:20:43.937 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.948+0000 7fc5be7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:43.937 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.948+0000 7fc5be7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:43.937 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.948+0000 7fc5be7fc640 10 monclient(hunting): my global_id is 4232 2024-03-29T11:20:43.937 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.948+0000 7fc5be7fc640 1 -- 172.21.15.86:0/2527566637 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5ac004eb0 con 0x7fc5c813f650 2024-03-29T11:20:43.937 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.949+0000 7fc5be7fc640 1 -- 172.21.15.86:0/2527566637 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3707766168 0 0) 0x7fc5c81d7080 con 0x7fc5c80a9d70 2024-03-29T11:20:43.937 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.949+0000 7fc5be7fc640 1 -- 172.21.15.86:0/2527566637 >> v1:172.21.15.86:6790/0 conn(0x7fc5c813f650 legacy=0x7fc5c80ab870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:43.937 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.949+0000 7fc5be7fc640 1 -- 172.21.15.86:0/2527566637 >> v1:172.21.15.86:6789/0 conn(0x7fc5c81457f0 legacy=0x7fc5c8137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:43.937 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.949+0000 7fc5be7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:20:43.937 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.949+0000 7fc5be7fc640 1 monclient: found mon.b 2024-03-29T11:20:43.937 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.949+0000 7fc5be7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:43.937 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.949+0000 7fc5be7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:20:43.938 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.949+0000 7fc5be7fc640 1 -- 172.21.15.86:0/2527566637 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c80ad070 con 0x7fc5c80a9d70 2024-03-29T11:20:43.938 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.949+0000 7fc5be7fc640 10 monclient: _finish_auth 0 2024-03-29T11:20:43.938 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.949+0000 7fc5be7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:20:43.938 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.949+0000 7fc5be7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:43.938 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.949+0000 7fc5cffdb640 5 monclient: authenticate success, global_id 4234 2024-03-29T11:20:43.938 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.949+0000 7fc5cffdb640 10 monclient: _renew_subs 2024-03-29T11:20:43.938 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.949+0000 7fc5cffdb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:20:43.938 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.949+0000 7fc5cffdb640 1 -- 172.21.15.86:0/2527566637 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5c8142ce0 con 0x7fc5c80a9d70 2024-03-29T11:20:43.938 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.949+0000 7fc5cffdb640 10 client.4234.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:43.938 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.949+0000 7fc5cffdb640 10 monclient: _renew_subs 2024-03-29T11:20:43.938 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.949+0000 7fc5cffdb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:20:43.938 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.949+0000 7fc5cffdb640 1 -- 172.21.15.86:0/2527566637 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5c813fda0 con 0x7fc5c80a9d70 2024-03-29T11:20:43.938 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.949+0000 7fc5cffdb640 1 librados: init done 2024-03-29T11:20:43.939 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.949+0000 7fc5be7fc640 1 -- 172.21.15.86:0/2527566637 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc5c40035f0 con 0x7fc5c80a9d70 2024-03-29T11:20:43.939 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.949+0000 7fc5be7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:43.939 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.949+0000 7fc5be7fc640 1 -- 172.21.15.86:0/2527566637 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc5c4003900 con 0x7fc5c80a9d70 2024-03-29T11:20:43.939 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.949+0000 7fc5be7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:43.939 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.949+0000 7fc5be7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:20:43.939 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.949+0000 7fc5be7fc640 10 monclient: dump: 2024-03-29T11:20:43.939 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:20:43.939 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:43.939 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:43.939 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:43.939 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:43.939 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:43.939 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:43.939 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:43.939 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:43.940 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:20:43.940 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.949+0000 7fc5bf7fe640 4 set_mon_vals no callback set 2024-03-29T11:20:43.940 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.950+0000 7fc5cffdb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:43.940 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.950+0000 7fc5cffdb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:43.940 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.950+0000 7fc5cffdb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:20:43.940 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.950+0000 7fc5cffdb640 1 -- 172.21.15.86:0/2527566637 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5b0005080 con 0x7fc5c80a9d70 2024-03-29T11:20:43.940 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.951+0000 7fc5be7fc640 1 -- 172.21.15.86:0/2527566637 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7fc5c40198d0 con 0x7fc5c80a9d70 2024-03-29T11:20:43.940 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.952+0000 7fc5be7fc640 1 -- 172.21.15.86:0/2527566637 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7fc5c4083740 con 0x7fc5c80a9d70 2024-03-29T11:20:43.940 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.952+0000 7fc5be7fc640 10 client.4234.objecter ms_dispatch 0x7fc5c8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:43.940 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.952+0000 7fc5be7fc640 3 client.4234.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:43.940 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.952+0000 7fc5be7fc640 3 client.4234.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:43.940 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.952+0000 7fc5be7fc640 20 client.4234.objecter dump_active .. 0 homeless 2024-03-29T11:20:43.940 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.952+0000 7fc5cd576640 10 monclient: get_auth_request con 0x7fc5ac06de00 auth_method 0 2024-03-29T11:20:43.940 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.952+0000 7fc5be7fc640 10 client.4234.objecter ms_handle_connect 0x7fc5ac06de00 2024-03-29T11:20:43.941 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.957+0000 7fc5be7fc640 1 -- 172.21.15.86:0/2527566637 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc5c404f020 con 0x7fc5c80a9d70 2024-03-29T11:20:43.941 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.957+0000 7fc5be7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:43.941 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:43.957+0000 7fc5be7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:44.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:44.341+0000 7fc5cffdb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:44.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:44.341+0000 7fc5cffdb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:44.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:44.341+0000 7fc5cffdb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:20:44.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:44.341+0000 7fc5cffdb640 1 -- 172.21.15.86:0/2527566637 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5b0005330 con 0x7fc5c80a9d70 2024-03-29T11:20:44.319 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:44.342+0000 7fc5be7fc640 1 -- 172.21.15.86:0/2527566637 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7fc5c4050f50 con 0x7fc5c80a9d70 2024-03-29T11:20:44.319 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:44.342+0000 7fc5be7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:44.319 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:44.342+0000 7fc5be7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:44.319 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:20:44.319 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:44.323 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:44.345+0000 7fc5cffdb640 10 librados: watch_flush enter 2024-03-29T11:20:44.323 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:44.345+0000 7fc5cffdb640 10 librados: watch_flush exit 2024-03-29T11:20:44.323 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:44.345+0000 7fc5cffdb640 20 client.4234.objecter shutdown clearing up homeless session... 2024-03-29T11:20:44.323 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:44.345+0000 7fc5cffdb640 10 client.4234.objecter successfully canceled tick 2024-03-29T11:20:44.323 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:44.346+0000 7fc5cffdb640 1 -- 172.21.15.86:0/2527566637 >> v1:172.21.15.196:6816/588110318 conn(0x7fc5ac06de00 legacy=0x7fc5ac0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:44.323 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:44.346+0000 7fc5cffdb640 10 monclient: shutdown 2024-03-29T11:20:44.323 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:44.346+0000 7fc5cffdb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:44.323 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:44.346+0000 7fc5cffdb640 1 -- 172.21.15.86:0/2527566637 >> v1:172.21.15.196:6789/0 conn(0x7fc5c80a9d70 legacy=0x7fc5c81d77f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:44.323 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:44.346+0000 7fc5cffdb640 1 -- 172.21.15.86:0/2527566637 shutdown_connections 2024-03-29T11:20:44.323 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:44.346+0000 7fc5cffdb640 1 -- 172.21.15.86:0/2527566637 >> 172.21.15.86:0/2527566637 conn(0x7fc5c8066a00 msgr2=0x7fc5c8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:44.323 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:44.346+0000 7fc5cffdb640 1 -- 172.21.15.86:0/2527566637 shutdown_connections 2024-03-29T11:20:44.323 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:44.346+0000 7fc5cffdb640 1 -- 172.21.15.86:0/2527566637 wait complete. 2024-03-29T11:20:44.324 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:44.347+0000 7fc5cffdb640 1 librados: shutdown 2024-03-29T11:20:44.338 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:51.145 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:51.296 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.320+0000 7f615d017640 10 monclient: get_monmap_and_config 2024-03-29T11:20:51.297 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.320+0000 7f615d017640 10 monclient: build_initial_monmap 2024-03-29T11:20:51.297 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.320+0000 7f615d017640 10 monclient: monmap: 2024-03-29T11:20:51.297 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:20:51.297 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:51.297 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:20:51.321212+0000 2024-03-29T11:20:51.297 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:20:51.321212+0000 2024-03-29T11:20:51.297 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:51.297 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:51.297 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:20:51.297 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:20:51.297 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:20:51.298 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:20:51.298 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.321+0000 7f615d017640 1 Processor -- start 2024-03-29T11:20:51.298 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.321+0000 7f615d017640 1 -- start start 2024-03-29T11:20:51.298 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.321+0000 7f615d017640 10 monclient: init 2024-03-29T11:20:51.302 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.322+0000 7f615d017640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:51.302 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.322+0000 7f615d017640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:51.302 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.322+0000 7f615d017640 10 monclient(hunting): picked mon.noname-b con 0x7f6158149c50 addr v1:172.21.15.86:6790/0 2024-03-29T11:20:51.302 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.322+0000 7f615d017640 10 monclient(hunting): picked mon.noname-c con 0x7f61580a9cc0 addr v1:172.21.15.196:6789/0 2024-03-29T11:20:51.302 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.322+0000 7f615d017640 10 monclient(hunting): picked mon.noname-a con 0x7f6158143d00 addr v1:172.21.15.86:6789/0 2024-03-29T11:20:51.302 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.322+0000 7f615d017640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61580ab9c0 con 0x7f6158143d00 2024-03-29T11:20:51.302 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.322+0000 7f615d017640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61580abc10 con 0x7f61580a9cc0 2024-03-29T11:20:51.302 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.322+0000 7f615d017640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61580aad20 con 0x7f6158149c50 2024-03-29T11:20:51.302 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.322+0000 7f615d017640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:51.303 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.322+0000 7f615d017640 10 monclient(hunting): authenticate will time out at 875.640512s 2024-03-29T11:20:51.303 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.322+0000 7f615759e640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f6158143d00 0x7f6158151d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:41180/0 (socket says 172.21.15.86:41180) 2024-03-29T11:20:51.303 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.322+0000 7f615759e640 1 -- 172.21.15.86:0/1149588920 learned_addr learned my addr 172.21.15.86:0/1149588920 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:20:51.303 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.323+0000 7f6155d9b640 1 -- 172.21.15.86:0/1149588920 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2320314142 0 0) 0x7f61580aad20 con 0x7f6158149c50 2024-03-29T11:20:51.303 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.323+0000 7f6155d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.303 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.323+0000 7f6155d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.303 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.323+0000 7f6155d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.303 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.323+0000 7f6155d9b640 10 monclient(hunting): my global_id is 4235 2024-03-29T11:20:51.303 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.323+0000 7f6155d9b640 1 -- 172.21.15.86:0/1149588920 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6140003520 con 0x7f6158149c50 2024-03-29T11:20:51.303 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.323+0000 7f6155d9b640 1 -- 172.21.15.86:0/1149588920 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1576018400 0 0) 0x7f61580abc10 con 0x7f61580a9cc0 2024-03-29T11:20:51.303 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.323+0000 7f6155d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.304 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.323+0000 7f6155d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.304 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.323+0000 7f6155d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.304 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.323+0000 7f6155d9b640 10 monclient(hunting): my global_id is 4237 2024-03-29T11:20:51.304 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.323+0000 7f6155d9b640 1 -- 172.21.15.86:0/1149588920 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f61580aad20 con 0x7f61580a9cc0 2024-03-29T11:20:51.304 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.323+0000 7f6155d9b640 1 -- 172.21.15.86:0/1149588920 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3652203817 0 0) 0x7f6140003520 con 0x7f6158149c50 2024-03-29T11:20:51.304 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.323+0000 7f6155d9b640 1 -- 172.21.15.86:0/1149588920 >> v1:172.21.15.196:6789/0 conn(0x7f61580a9cc0 legacy=0x7f61581435f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.304 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.323+0000 7f6155d9b640 1 -- 172.21.15.86:0/1149588920 >> v1:172.21.15.86:6789/0 conn(0x7f6158143d00 legacy=0x7f6158151d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.304 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.324+0000 7f6155d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:20:51.304 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.324+0000 7f6155d9b640 1 monclient: found mon.noname-b 2024-03-29T11:20:51.305 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.324+0000 7f6155d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:51.305 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.324+0000 7f6155d9b640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:20:51.305 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.324+0000 7f6155d9b640 1 -- 172.21.15.86:0/1149588920 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61580aa980 con 0x7f6158149c50 2024-03-29T11:20:51.305 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.324+0000 7f6155d9b640 10 monclient: _finish_auth 0 2024-03-29T11:20:51.305 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.324+0000 7f6155d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:20:51.305 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.324+0000 7f6155d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:51.305 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.324+0000 7f6155d9b640 1 -- 172.21.15.86:0/1149588920 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f6148001690 con 0x7f6158149c50 2024-03-29T11:20:51.305 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.324+0000 7f6155d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.305 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.324+0000 7f6155d9b640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:51.305 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.324+0000 7f6155d9b640 10 monclient: dump: 2024-03-29T11:20:51.305 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:20:51.305 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:51.305 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:51.305 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:51.306 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.306 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:51.306 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:51.306 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:51.306 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:51.306 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:20:51.306 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.324+0000 7f615d017640 5 monclient: authenticate success, global_id 4235 2024-03-29T11:20:51.306 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.324+0000 7f615d017640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:51.306 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.325+0000 7f6155d9b640 1 -- 172.21.15.86:0/1149588920 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f61480017f0 con 0x7f6158149c50 2024-03-29T11:20:51.306 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.325+0000 7f6155d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:51.306 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.325+0000 7f615d017640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:51.306 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.325+0000 7f615d017640 4 set_mon_vals no callback set 2024-03-29T11:20:51.306 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.325+0000 7f6155d9b640 1 -- 172.21.15.86:0/1149588920 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f6148001c40 con 0x7f6158149c50 2024-03-29T11:20:51.306 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.325+0000 7f6155d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.307 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.325+0000 7f6155d9b640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:51.307 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.325+0000 7f6155d9b640 10 monclient: dump: 2024-03-29T11:20:51.307 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:20:51.307 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:51.307 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:51.307 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:51.307 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.307 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:51.307 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:51.307 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:51.307 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:51.308 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:20:51.314 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.326+0000 7f615d017640 10 monclient: shutdown 2024-03-29T11:20:51.314 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.326+0000 7f615d017640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:51.314 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.326+0000 7f615d017640 1 -- 172.21.15.86:0/1149588920 >> v1:172.21.15.86:6790/0 conn(0x7f6158149c50 legacy=0x7f61580ae650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.314 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.326+0000 7f615d017640 1 -- 172.21.15.86:0/1149588920 shutdown_connections 2024-03-29T11:20:51.314 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.326+0000 7f615d017640 1 -- 172.21.15.86:0/1149588920 >> 172.21.15.86:0/1149588920 conn(0x7f61580b3c50 msgr2=0x7f61580b4060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:51.314 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.326+0000 7f615d017640 1 -- 172.21.15.86:0/1149588920 shutdown_connections 2024-03-29T11:20:51.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.326+0000 7f615d017640 1 -- 172.21.15.86:0/1149588920 wait complete. 2024-03-29T11:20:51.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.327+0000 7f615d017640 10 monclient: build_initial_monmap 2024-03-29T11:20:51.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.327+0000 7f615d017640 10 monclient: monmap: 2024-03-29T11:20:51.315 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:20:51.315 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:51.315 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:20:51.315 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:20:51.315 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:51.315 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:51.315 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:20:51.315 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:20:51.315 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:20:51.315 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:20:51.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.328+0000 7f615d017640 1 librados: starting msgr at 2024-03-29T11:20:51.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.328+0000 7f615d017640 1 librados: starting objecter 2024-03-29T11:20:51.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.328+0000 7f615d017640 1 Processor -- start 2024-03-29T11:20:51.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.328+0000 7f615d017640 1 -- start start 2024-03-29T11:20:51.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.328+0000 7f615d017640 1 librados: setting wanted keys 2024-03-29T11:20:51.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.328+0000 7f615d017640 1 librados: calling monclient init 2024-03-29T11:20:51.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.328+0000 7f615d017640 10 monclient: init 2024-03-29T11:20:51.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.329+0000 7f615d017640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:51.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.329+0000 7f615d017640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:51.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.329+0000 7f615d017640 10 monclient(hunting): picked mon.noname-c con 0x7f61580a9cc0 addr v1:172.21.15.86:6790/0 2024-03-29T11:20:51.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.329+0000 7f615d017640 10 monclient(hunting): picked mon.noname-b con 0x7f6158143d00 addr v1:172.21.15.196:6789/0 2024-03-29T11:20:51.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.329+0000 7f6156d9d640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f61580a9cc0 0x7f6158071130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:57438/0 (socket says 172.21.15.86:57438) 2024-03-29T11:20:51.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.329+0000 7f615659c640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f6158143d00 0x7f615813e530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:44740/0 (socket says 172.21.15.86:44740) 2024-03-29T11:20:51.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.329+0000 7f6156d9d640 1 -- 172.21.15.86:0/2152862534 learned_addr learned my addr 172.21.15.86:0/2152862534 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:20:51.317 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.329+0000 7f615d017640 10 monclient(hunting): picked mon.noname-a con 0x7f6158149c50 addr v1:172.21.15.86:6789/0 2024-03-29T11:20:51.317 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.329+0000 7f615d017640 1 -- 172.21.15.86:0/2152862534 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f615813f350 con 0x7f6158149c50 2024-03-29T11:20:51.317 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.329+0000 7f613f7fe640 10 client.?.objecter ms_handle_connect 0x7f61580a9cc0 2024-03-29T11:20:51.317 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.329+0000 7f613f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:51.317 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.329+0000 7f615d017640 1 -- 172.21.15.86:0/2152862534 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61580706e0 con 0x7f6158143d00 2024-03-29T11:20:51.317 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.329+0000 7f615d017640 1 -- 172.21.15.86:0/2152862534 --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f61580709c0 con 0x7f61580a9cc0 2024-03-29T11:20:51.317 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.329+0000 7f615d017640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:51.317 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.329+0000 7f613f7fe640 10 client.?.objecter ms_handle_connect 0x7f6158149c50 2024-03-29T11:20:51.317 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.329+0000 7f615d017640 10 monclient(hunting): authenticate will time out at 875.647993s 2024-03-29T11:20:51.317 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.330+0000 7f613f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:51.317 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.330+0000 7f613f7fe640 10 client.?.objecter ms_handle_connect 0x7f6158143d00 2024-03-29T11:20:51.317 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.330+0000 7f613f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:51.317 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.330+0000 7f613f7fe640 1 -- 172.21.15.86:0/2152862534 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1318963994 0 0) 0x7f615813f350 con 0x7f6158149c50 2024-03-29T11:20:51.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.330+0000 7f613f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.330+0000 7f613f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.330+0000 7f613f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.330+0000 7f613f7fe640 10 monclient(hunting): my global_id is 4248 2024-03-29T11:20:51.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.330+0000 7f613f7fe640 1 -- 172.21.15.86:0/2152862534 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6144004eb0 con 0x7f6158149c50 2024-03-29T11:20:51.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.330+0000 7f613f7fe640 1 -- 172.21.15.86:0/2152862534 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 450979670 0 0) 0x7f61580709c0 con 0x7f61580a9cc0 2024-03-29T11:20:51.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.330+0000 7f613f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:20:51.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.330+0000 7f613f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:51.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.330+0000 7f613f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:51.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.330+0000 7f613f7fe640 10 monclient(hunting): my global_id is 4238 2024-03-29T11:20:51.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.330+0000 7f613f7fe640 1 -- 172.21.15.86:0/2152862534 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f615813f350 con 0x7f61580a9cc0 2024-03-29T11:20:51.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.330+0000 7f613f7fe640 1 -- 172.21.15.86:0/2152862534 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4187187284 0 0) 0x7f6144004eb0 con 0x7f6158149c50 2024-03-29T11:20:51.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.330+0000 7f613f7fe640 1 -- 172.21.15.86:0/2152862534 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f61580709c0 con 0x7f6158149c50 2024-03-29T11:20:51.319 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.330+0000 7f613f7fe640 1 -- 172.21.15.86:0/2152862534 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f614c0018b0 con 0x7f6158149c50 2024-03-29T11:20:51.319 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.330+0000 7f613f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.319 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.330+0000 7f613f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:20:51.319 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.330+0000 7f613f7fe640 10 monclient(hunting): dump: 2024-03-29T11:20:51.319 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:20:51.319 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:51.319 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:51.319 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:51.319 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.319 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:51.319 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:51.319 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:51.319 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:51.319 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:20:51.319 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.330+0000 7f613f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:51.320 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.331+0000 7f613f7fe640 1 -- 172.21.15.86:0/2152862534 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3008241234 0 0) 0x7f615813f350 con 0x7f61580a9cc0 2024-03-29T11:20:51.320 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.331+0000 7f613f7fe640 1 -- 172.21.15.86:0/2152862534 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6144004eb0 con 0x7f61580a9cc0 2024-03-29T11:20:51.320 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.331+0000 7f613f7fe640 1 -- 172.21.15.86:0/2152862534 <== mon.0 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 406558848 0 0) 0x7f61580709c0 con 0x7f6158149c50 2024-03-29T11:20:51.320 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.331+0000 7f613f7fe640 1 -- 172.21.15.86:0/2152862534 >> v1:172.21.15.86:6790/0 conn(0x7f61580a9cc0 legacy=0x7f6158071130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.320 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.331+0000 7f613f7fe640 1 -- 172.21.15.86:0/2152862534 >> v1:172.21.15.196:6789/0 conn(0x7f6158143d00 legacy=0x7f615813e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.320 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.331+0000 7f613f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:20:51.320 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.331+0000 7f613f7fe640 1 monclient: found mon.a 2024-03-29T11:20:51.320 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.331+0000 7f613f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:51.320 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.331+0000 7f613f7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:20:51.320 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.331+0000 7f613f7fe640 1 -- 172.21.15.86:0/2152862534 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f615806f580 con 0x7f6158149c50 2024-03-29T11:20:51.320 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.331+0000 7f613f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:20:51.320 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.331+0000 7f613f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:20:51.320 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.331+0000 7f613f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:51.321 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.331+0000 7f615d017640 5 monclient: authenticate success, global_id 4248 2024-03-29T11:20:51.321 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.331+0000 7f615d017640 10 monclient: _renew_subs 2024-03-29T11:20:51.321 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.331+0000 7f615d017640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:20:51.321 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.331+0000 7f615d017640 1 -- 172.21.15.86:0/2152862534 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f615806faa0 con 0x7f6158149c50 2024-03-29T11:20:51.321 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.331+0000 7f615d017640 10 client.4248.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:51.321 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.331+0000 7f615d017640 10 monclient: _renew_subs 2024-03-29T11:20:51.321 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.331+0000 7f615d017640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:20:51.321 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.331+0000 7f615d017640 1 -- 172.21.15.86:0/2152862534 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f615806ea90 con 0x7f6158149c50 2024-03-29T11:20:51.321 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.331+0000 7f615d017640 1 librados: init done 2024-03-29T11:20:51.321 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.332+0000 7f613f7fe640 1 -- 172.21.15.86:0/2152862534 <== mon.0 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f614c001d40 con 0x7f6158149c50 2024-03-29T11:20:51.321 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.332+0000 7f613f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:51.321 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.332+0000 7f613f7fe640 1 -- 172.21.15.86:0/2152862534 <== mon.0 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f614c004980 con 0x7f6158149c50 2024-03-29T11:20:51.322 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.332+0000 7f613f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:51.322 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.332+0000 7f615559a640 4 set_mon_vals no callback set 2024-03-29T11:20:51.322 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.332+0000 7f613f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:20:51.322 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.332+0000 7f613f7fe640 10 monclient: dump: 2024-03-29T11:20:51.322 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:20:51.322 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:51.322 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:51.322 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:51.322 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:51.322 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:51.322 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:51.323 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:51.323 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:51.323 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:20:51.323 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.333+0000 7f615d017640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:51.323 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.333+0000 7f615d017640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:51.323 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.333+0000 7f615d017640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:20:51.323 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.333+0000 7f615d017640 1 -- 172.21.15.86:0/2152862534 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6138005080 con 0x7f6158149c50 2024-03-29T11:20:51.323 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.334+0000 7f613f7fe640 1 -- 172.21.15.86:0/2152862534 <== mon.0 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f614c01a950 con 0x7f6158149c50 2024-03-29T11:20:51.323 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.335+0000 7f613f7fe640 1 -- 172.21.15.86:0/2152862534 <== mon.0 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f614c0848a0 con 0x7f6158149c50 2024-03-29T11:20:51.323 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.335+0000 7f613f7fe640 10 client.4248.objecter ms_dispatch 0x7f6158000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:51.323 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.335+0000 7f613f7fe640 3 client.4248.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:51.323 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.335+0000 7f613f7fe640 3 client.4248.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:51.323 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.335+0000 7f613f7fe640 20 client.4248.objecter dump_active .. 0 homeless 2024-03-29T11:20:51.323 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.335+0000 7f6156d9d640 10 monclient: get_auth_request con 0x7f614406dea0 auth_method 0 2024-03-29T11:20:51.323 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.336+0000 7f613f7fe640 10 client.4248.objecter ms_handle_connect 0x7f614406dea0 2024-03-29T11:20:51.324 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.344+0000 7f613f7fe640 1 -- 172.21.15.86:0/2152862534 <== mon.0 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f614c04ffb0 con 0x7f6158149c50 2024-03-29T11:20:51.324 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.344+0000 7f613f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:51.324 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.344+0000 7f613f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:51.648 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.671+0000 7f615d017640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:51.648 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.671+0000 7f615d017640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:51.648 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.671+0000 7f615d017640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:20:51.648 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.671+0000 7f615d017640 1 -- 172.21.15.86:0/2152862534 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6138005330 con 0x7f6158149c50 2024-03-29T11:20:51.649 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.672+0000 7f613f7fe640 1 -- 172.21.15.86:0/2152862534 <== mon.0 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f614c0520b0 con 0x7f6158149c50 2024-03-29T11:20:51.649 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.672+0000 7f613f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:51.649 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.672+0000 7f613f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:51.649 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:20:51.649 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:51.652 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.675+0000 7f615d017640 10 librados: watch_flush enter 2024-03-29T11:20:51.652 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.675+0000 7f615d017640 10 librados: watch_flush exit 2024-03-29T11:20:51.652 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.675+0000 7f615d017640 20 client.4248.objecter shutdown clearing up homeless session... 2024-03-29T11:20:51.652 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.675+0000 7f615d017640 10 client.4248.objecter successfully canceled tick 2024-03-29T11:20:51.652 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.675+0000 7f615d017640 1 -- 172.21.15.86:0/2152862534 >> v1:172.21.15.196:6816/588110318 conn(0x7f614406dea0 legacy=0x7f6144070360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.652 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.675+0000 7f615d017640 10 monclient: shutdown 2024-03-29T11:20:51.652 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.675+0000 7f615d017640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:51.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.675+0000 7f615d017640 1 -- 172.21.15.86:0/2152862534 >> v1:172.21.15.86:6789/0 conn(0x7f6158149c50 legacy=0x7f615813ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:51.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.675+0000 7f615d017640 1 -- 172.21.15.86:0/2152862534 shutdown_connections 2024-03-29T11:20:51.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.675+0000 7f615d017640 1 -- 172.21.15.86:0/2152862534 >> 172.21.15.86:0/2152862534 conn(0x7f61580b3c50 msgr2=0x7f6158142aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:51.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.675+0000 7f615d017640 1 -- 172.21.15.86:0/2152862534 shutdown_connections 2024-03-29T11:20:51.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.675+0000 7f615d017640 1 -- 172.21.15.86:0/2152862534 wait complete. 2024-03-29T11:20:51.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:51.675+0000 7f615d017640 1 librados: shutdown 2024-03-29T11:20:51.665 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:20:58.472 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:20:58.606 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.629+0000 7f83d9774640 10 monclient: get_monmap_and_config 2024-03-29T11:20:58.607 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.629+0000 7f83d9774640 10 monclient: build_initial_monmap 2024-03-29T11:20:58.607 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.629+0000 7f83d9774640 10 monclient: monmap: 2024-03-29T11:20:58.607 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:20:58.607 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:58.607 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:20:58.631126+0000 2024-03-29T11:20:58.607 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:20:58.631126+0000 2024-03-29T11:20:58.607 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:58.607 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:58.607 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:20:58.607 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:20:58.607 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:20:58.607 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:20:58.610 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.630+0000 7f83d9774640 1 Processor -- start 2024-03-29T11:20:58.610 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.630+0000 7f83d9774640 1 -- start start 2024-03-29T11:20:58.610 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.630+0000 7f83d9774640 10 monclient: init 2024-03-29T11:20:58.610 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.631+0000 7f83d9774640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:58.610 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.631+0000 7f83d9774640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:20:58.610 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.631+0000 7f83d9774640 10 monclient(hunting): picked mon.noname-b con 0x7f83cc0345e0 addr v1:172.21.15.86:6790/0 2024-03-29T11:20:58.610 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.631+0000 7f83d9774640 10 monclient(hunting): picked mon.noname-c con 0x7f83cc0349c0 addr v1:172.21.15.196:6789/0 2024-03-29T11:20:58.610 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.631+0000 7f83d9774640 10 monclient(hunting): picked mon.noname-a con 0x7f83cc02b570 addr v1:172.21.15.86:6789/0 2024-03-29T11:20:58.610 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.631+0000 7f83d9774640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83cc032ef0 con 0x7f83cc02b570 2024-03-29T11:20:58.611 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.631+0000 7f83d9774640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83cc0be170 con 0x7f83cc0349c0 2024-03-29T11:20:58.611 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.631+0000 7f83d9774640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83cc02c720 con 0x7f83cc0345e0 2024-03-29T11:20:58.611 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.631+0000 7f83d9774640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:58.611 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.631+0000 7f83d9774640 10 monclient(hunting): authenticate will time out at 882.950148s 2024-03-29T11:20:58.611 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.631+0000 7f83d3fff640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f83cc0345e0 0x7f83cc0bda60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:48324/0 (socket says 172.21.15.86:48324) 2024-03-29T11:20:58.611 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.631+0000 7f83d3fff640 1 -- 172.21.15.86:0/33644780 learned_addr learned my addr 172.21.15.86:0/33644780 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:20:58.611 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.631+0000 7f83d2ffd640 1 -- 172.21.15.86:0/33644780 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3271274979 0 0) 0x7f83cc02c720 con 0x7f83cc0345e0 2024-03-29T11:20:58.611 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.631+0000 7f83d2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:20:58.611 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.631+0000 7f83d2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:58.611 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.631+0000 7f83d2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:58.611 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.631+0000 7f83d2ffd640 10 monclient(hunting): my global_id is 4241 2024-03-29T11:20:58.611 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.631+0000 7f83d2ffd640 1 -- 172.21.15.86:0/33644780 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83c4003520 con 0x7f83cc0345e0 2024-03-29T11:20:58.611 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.631+0000 7f83d2ffd640 1 -- 172.21.15.86:0/33644780 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1163908871 0 0) 0x7f83c4003520 con 0x7f83cc0345e0 2024-03-29T11:20:58.611 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.631+0000 7f83d2ffd640 1 -- 172.21.15.86:0/33644780 >> v1:172.21.15.196:6789/0 conn(0x7f83cc0349c0 legacy=0x7f83cc0c7260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:20:58.612 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.631+0000 7f83d2ffd640 1 -- 172.21.15.86:0/33644780 >> v1:172.21.15.86:6789/0 conn(0x7f83cc02b570 legacy=0x7f83cc0c7970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:20:58.615 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.634+0000 7f83d2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:20:58.615 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.634+0000 7f83d2ffd640 1 monclient: found mon.noname-b 2024-03-29T11:20:58.615 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.634+0000 7f83d2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:58.616 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.634+0000 7f83d2ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:20:58.616 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.634+0000 7f83d2ffd640 1 -- 172.21.15.86:0/33644780 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83cc02c990 con 0x7f83cc0345e0 2024-03-29T11:20:58.616 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.634+0000 7f83d2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:20:58.616 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.634+0000 7f83d2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:20:58.616 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.634+0000 7f83d2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:58.616 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.634+0000 7f83d2ffd640 1 -- 172.21.15.86:0/33644780 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f83c8001670 con 0x7f83cc0345e0 2024-03-29T11:20:58.616 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.634+0000 7f83d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:58.616 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.634+0000 7f83d2ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:20:58.616 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.634+0000 7f83d2ffd640 10 monclient: dump: 2024-03-29T11:20:58.616 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:20:58.616 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:58.616 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:58.616 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:58.616 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:58.616 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:58.617 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:58.617 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:58.617 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:58.617 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:20:58.617 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.634+0000 7f83d9774640 5 monclient: authenticate success, global_id 4241 2024-03-29T11:20:58.617 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.634+0000 7f83d9774640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:20:58.617 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.634+0000 7f83d2ffd640 1 -- 172.21.15.86:0/33644780 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83c80018d0 con 0x7f83cc0345e0 2024-03-29T11:20:58.617 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.634+0000 7f83d2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:58.617 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.634+0000 7f83d2ffd640 1 -- 172.21.15.86:0/33644780 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f83c8001d20 con 0x7f83cc0345e0 2024-03-29T11:20:58.617 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.634+0000 7f83d9774640 10 monclient: get_monmap_and_config success 2024-03-29T11:20:58.617 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.634+0000 7f83d9774640 4 set_mon_vals no callback set 2024-03-29T11:20:58.617 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.634+0000 7f83d2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:58.617 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.634+0000 7f83d2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:58.617 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.634+0000 7f83d2ffd640 10 monclient: dump: 2024-03-29T11:20:58.618 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:20:58.618 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:58.618 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:58.618 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:58.618 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:58.618 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:58.618 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:58.618 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:58.618 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:58.618 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:20:58.618 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.635+0000 7f83d9774640 10 monclient: shutdown 2024-03-29T11:20:58.618 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.635+0000 7f83d9774640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:58.618 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.635+0000 7f83d9774640 1 -- 172.21.15.86:0/33644780 >> v1:172.21.15.86:6790/0 conn(0x7f83cc0345e0 legacy=0x7f83cc0bda60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:58.618 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.635+0000 7f83d9774640 1 -- 172.21.15.86:0/33644780 shutdown_connections 2024-03-29T11:20:58.619 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.635+0000 7f83d9774640 1 -- 172.21.15.86:0/33644780 >> 172.21.15.86:0/33644780 conn(0x7f83cc0058f0 msgr2=0x7f83cc005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:58.619 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.635+0000 7f83d9774640 1 -- 172.21.15.86:0/33644780 shutdown_connections 2024-03-29T11:20:58.619 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.636+0000 7f83d9774640 1 -- 172.21.15.86:0/33644780 wait complete. 2024-03-29T11:20:58.619 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.637+0000 7f83d9774640 10 monclient: build_initial_monmap 2024-03-29T11:20:58.619 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.637+0000 7f83d9774640 10 monclient: monmap: 2024-03-29T11:20:58.619 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:20:58.619 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:20:58.619 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:20:58.619 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:20:58.619 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:20:58.619 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:58.619 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:20:58.619 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:20:58.619 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:20:58.619 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:20:58.620 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.637+0000 7f83d9774640 1 librados: starting msgr at 2024-03-29T11:20:58.620 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.637+0000 7f83d9774640 1 librados: starting objecter 2024-03-29T11:20:58.620 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.637+0000 7f83d9774640 1 Processor -- start 2024-03-29T11:20:58.620 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.638+0000 7f83d9774640 1 -- start start 2024-03-29T11:20:58.620 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.638+0000 7f83d9774640 1 librados: setting wanted keys 2024-03-29T11:20:58.620 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.638+0000 7f83d9774640 1 librados: calling monclient init 2024-03-29T11:20:58.620 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.638+0000 7f83d9774640 10 monclient: init 2024-03-29T11:20:58.626 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.639+0000 7f83d9774640 10 monclient: _reopen_session rank -1 2024-03-29T11:20:58.626 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.639+0000 7f83d9774640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:20:58.626 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.639+0000 7f83d9774640 10 monclient(hunting): picked mon.noname-c con 0x7f83cc02b570 addr v1:172.21.15.86:6790/0 2024-03-29T11:20:58.626 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.639+0000 7f83d9774640 10 monclient(hunting): picked mon.noname-a con 0x7f83cc0c01a0 addr v1:172.21.15.86:6789/0 2024-03-29T11:20:58.626 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.639+0000 7f83d9774640 10 monclient(hunting): picked mon.noname-b con 0x7f83cc0bc510 addr v1:172.21.15.196:6789/0 2024-03-29T11:20:58.626 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.639+0000 7f83d9774640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83cc031fd0 con 0x7f83cc0c01a0 2024-03-29T11:20:58.627 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.639+0000 7f83d9774640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83cc15c7d0 con 0x7f83cc0bc510 2024-03-29T11:20:58.627 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.639+0000 7f83d3fff640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f83cc02b570 0x7f83cc15bd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:51040/0 (socket says 172.21.15.86:51040) 2024-03-29T11:20:58.627 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.639+0000 7f83d3fff640 1 -- 172.21.15.86:0/2788398162 learned_addr learned my addr 172.21.15.86:0/2788398162 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:20:58.627 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.639+0000 7f83d9774640 1 -- 172.21.15.86:0/2788398162 --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f83cc15c460 con 0x7f83cc02b570 2024-03-29T11:20:58.627 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.639+0000 7f83d9774640 10 monclient(hunting): _renew_subs 2024-03-29T11:20:58.627 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.639+0000 7f83d9774640 10 monclient(hunting): authenticate will time out at 882.958364s 2024-03-29T11:20:58.627 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.639+0000 7f83d0ff9640 10 client.?.objecter ms_handle_connect 0x7f83cc02b570 2024-03-29T11:20:58.627 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.639+0000 7f83d0ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:20:58.627 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.639+0000 7f83d0ff9640 1 -- 172.21.15.86:0/2788398162 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4100946164 0 0) 0x7f83cc15c460 con 0x7f83cc02b570 2024-03-29T11:20:58.627 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.639+0000 7f83d0ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:20:58.627 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.639+0000 7f83d0ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:20:58.627 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.639+0000 7f83d0ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:20:58.627 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.639+0000 7f83d0ff9640 10 monclient(hunting): my global_id is 4244 2024-03-29T11:20:58.627 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.639+0000 7f83d0ff9640 1 -- 172.21.15.86:0/2788398162 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f83b8004eb0 con 0x7f83cc02b570 2024-03-29T11:20:58.628 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.640+0000 7f83d0ff9640 1 -- 172.21.15.86:0/2788398162 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3167387358 0 0) 0x7f83b8004eb0 con 0x7f83cc02b570 2024-03-29T11:20:58.628 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.640+0000 7f83d0ff9640 1 -- 172.21.15.86:0/2788398162 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f83cc15c460 con 0x7f83cc02b570 2024-03-29T11:20:58.628 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.640+0000 7f83d0ff9640 1 -- 172.21.15.86:0/2788398162 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f83c8001630 con 0x7f83cc02b570 2024-03-29T11:20:58.628 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.640+0000 7f83d0ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:58.628 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.640+0000 7f83d0ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:20:58.628 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.640+0000 7f83d0ff9640 10 monclient(hunting): dump: 2024-03-29T11:20:58.628 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:20:58.628 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:58.628 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:58.628 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:58.628 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:58.628 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:58.628 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:58.628 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:58.628 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:58.629 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:20:58.629 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.640+0000 7f83d0ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:20:58.629 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.640+0000 7f83d0ff9640 1 -- 172.21.15.86:0/2788398162 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2684914334 0 0) 0x7f83cc15c460 con 0x7f83cc02b570 2024-03-29T11:20:58.629 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.640+0000 7f83d0ff9640 1 -- 172.21.15.86:0/2788398162 >> v1:172.21.15.196:6789/0 conn(0x7f83cc0bc510 legacy=0x7f83cc0c1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:58.629 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.640+0000 7f83d0ff9640 1 -- 172.21.15.86:0/2788398162 >> v1:172.21.15.86:6789/0 conn(0x7f83cc0c01a0 legacy=0x7f83cc0bbe00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:20:58.629 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.642+0000 7f83d0ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:20:58.629 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.642+0000 7f83d0ff9640 1 monclient: found mon.c 2024-03-29T11:20:58.629 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.642+0000 7f83d0ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:20:58.629 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.642+0000 7f83d0ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:20:58.629 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.642+0000 7f83d0ff9640 1 -- 172.21.15.86:0/2788398162 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83cc0c21f0 con 0x7f83cc02b570 2024-03-29T11:20:58.629 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.642+0000 7f83d0ff9640 10 monclient: _finish_auth 0 2024-03-29T11:20:58.629 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.642+0000 7f83d0ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:20:58.629 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.642+0000 7f83d0ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:20:58.629 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.643+0000 7f83d9774640 5 monclient: authenticate success, global_id 4244 2024-03-29T11:20:58.630 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.643+0000 7f83d9774640 10 monclient: _renew_subs 2024-03-29T11:20:58.630 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.643+0000 7f83d9774640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:20:58.630 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.643+0000 7f83d9774640 1 -- 172.21.15.86:0/2788398162 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83cc0c2440 con 0x7f83cc02b570 2024-03-29T11:20:58.630 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.643+0000 7f83d9774640 10 client.4244.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:20:58.630 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.643+0000 7f83d9774640 10 monclient: _renew_subs 2024-03-29T11:20:58.630 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.643+0000 7f83d9774640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:20:58.630 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.643+0000 7f83d9774640 1 -- 172.21.15.86:0/2788398162 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f83cc1652c0 con 0x7f83cc02b570 2024-03-29T11:20:58.630 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.643+0000 7f83d9774640 1 librados: init done 2024-03-29T11:20:58.630 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.643+0000 7f83d0ff9640 1 -- 172.21.15.86:0/2788398162 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f83c8004c70 con 0x7f83cc02b570 2024-03-29T11:20:58.630 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.643+0000 7f83d0ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:20:58.630 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.643+0000 7f83d0ff9640 1 -- 172.21.15.86:0/2788398162 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f83c8004f80 con 0x7f83cc02b570 2024-03-29T11:20:58.630 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.643+0000 7f83d0ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:20:58.630 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.643+0000 7f83d0ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:20:58.630 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.643+0000 7f83d0ff9640 10 monclient: dump: 2024-03-29T11:20:58.631 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:20:58.631 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:20:58.631 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:58.631 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:20:58.631 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:20:58.631 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:20:58.631 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:20:58.631 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:20:58.631 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:20:58.631 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:20:58.631 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.643+0000 7f83d27fc640 4 set_mon_vals no callback set 2024-03-29T11:20:58.631 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.645+0000 7f83d0ff9640 1 -- 172.21.15.86:0/2788398162 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f83c801af50 con 0x7f83cc02b570 2024-03-29T11:20:58.631 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.645+0000 7f83d9774640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:58.631 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.645+0000 7f83d9774640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:58.631 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.645+0000 7f83d9774640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:20:58.632 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.646+0000 7f83d0ff9640 1 -- 172.21.15.86:0/2788398162 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f83c8084d20 con 0x7f83cc02b570 2024-03-29T11:20:58.632 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.646+0000 7f83d0ff9640 10 client.4244.objecter ms_dispatch 0x7f83d4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:20:58.632 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.646+0000 7f83d0ff9640 3 client.4244.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:20:58.632 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.646+0000 7f83d0ff9640 3 client.4244.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:20:58.632 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.646+0000 7f83d9774640 1 -- 172.21.15.86:0/2788398162 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83b4005080 con 0x7f83cc02b570 2024-03-29T11:20:58.632 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.646+0000 7f83d0ff9640 20 client.4244.objecter dump_active .. 0 homeless 2024-03-29T11:20:58.632 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.646+0000 7f83d37fe640 10 monclient: get_auth_request con 0x7f83b806dcd0 auth_method 0 2024-03-29T11:20:58.632 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.646+0000 7f83d0ff9640 10 client.4244.objecter ms_handle_connect 0x7f83b806dcd0 2024-03-29T11:20:58.633 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.653+0000 7f83d0ff9640 1 -- 172.21.15.86:0/2788398162 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f83c8050600 con 0x7f83cc02b570 2024-03-29T11:20:58.633 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.653+0000 7f83d0ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:20:58.633 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.653+0000 7f83d0ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:20:58.948 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.970+0000 7f83d9774640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:58.948 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.970+0000 7f83d9774640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:58.948 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.970+0000 7f83d9774640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:20:58.948 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.970+0000 7f83d9774640 1 -- 172.21.15.86:0/2788398162 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f83b4005330 con 0x7f83cc02b570 2024-03-29T11:20:58.949 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.971+0000 7f83d0ff9640 1 -- 172.21.15.86:0/2788398162 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f83c8052530 con 0x7f83cc02b570 2024-03-29T11:20:58.949 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.971+0000 7f83d0ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:20:58.949 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.971+0000 7f83d0ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:20:58.949 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:20:58.949 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:20:58.952 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.974+0000 7f83d9774640 10 librados: watch_flush enter 2024-03-29T11:20:58.952 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.974+0000 7f83d9774640 10 librados: watch_flush exit 2024-03-29T11:20:58.952 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.974+0000 7f83d9774640 20 client.4244.objecter shutdown clearing up homeless session... 2024-03-29T11:20:58.952 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.974+0000 7f83d9774640 10 client.4244.objecter successfully canceled tick 2024-03-29T11:20:58.952 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.974+0000 7f83d9774640 1 -- 172.21.15.86:0/2788398162 >> v1:172.21.15.196:6816/588110318 conn(0x7f83b806dcd0 legacy=0x7f83b8070190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:58.952 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.974+0000 7f83d9774640 10 monclient: shutdown 2024-03-29T11:20:58.952 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.974+0000 7f83d9774640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:20:58.952 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.974+0000 7f83d9774640 1 -- 172.21.15.86:0/2788398162 >> v1:172.21.15.86:6790/0 conn(0x7f83cc02b570 legacy=0x7f83cc15bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:20:58.953 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.975+0000 7f83d9774640 1 -- 172.21.15.86:0/2788398162 shutdown_connections 2024-03-29T11:20:58.953 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.975+0000 7f83d9774640 1 -- 172.21.15.86:0/2788398162 >> 172.21.15.86:0/2788398162 conn(0x7f83cc0058f0 msgr2=0x7f83cc033d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:20:58.953 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.975+0000 7f83d9774640 1 -- 172.21.15.86:0/2788398162 shutdown_connections 2024-03-29T11:20:58.953 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.975+0000 7f83d9774640 1 -- 172.21.15.86:0/2788398162 wait complete. 2024-03-29T11:20:58.953 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:20:58.975+0000 7f83d9774640 1 librados: shutdown 2024-03-29T11:20:58.967 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:05.774 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:05.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.948+0000 7fb451196640 10 monclient: get_monmap_and_config 2024-03-29T11:21:05.925 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.948+0000 7fb451196640 10 monclient: build_initial_monmap 2024-03-29T11:21:05.925 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.948+0000 7fb451196640 10 monclient: monmap: 2024-03-29T11:21:05.925 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:21:05.925 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:05.925 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:21:05.949124+0000 2024-03-29T11:21:05.925 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:21:05.949124+0000 2024-03-29T11:21:05.925 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:05.925 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:05.925 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:21:05.925 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:21:05.925 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:21:05.925 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:05.927 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.948+0000 7fb451196640 1 Processor -- start 2024-03-29T11:21:05.927 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.949+0000 7fb451196640 1 -- start start 2024-03-29T11:21:05.927 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.949+0000 7fb451196640 10 monclient: init 2024-03-29T11:21:05.927 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.949+0000 7fb451196640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:05.927 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.949+0000 7fb451196640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:05.927 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.949+0000 7fb451196640 10 monclient(hunting): picked mon.noname-b con 0x7fb44c1457f0 addr v1:172.21.15.86:6790/0 2024-03-29T11:21:05.927 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.949+0000 7fb451196640 10 monclient(hunting): picked mon.noname-c con 0x7fb44c13f650 addr v1:172.21.15.196:6789/0 2024-03-29T11:21:05.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.949+0000 7fb451196640 10 monclient(hunting): picked mon.noname-a con 0x7fb44c0a9d70 addr v1:172.21.15.86:6789/0 2024-03-29T11:21:05.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.950+0000 7fb451196640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb44c0aaef0 con 0x7fb44c0a9d70 2024-03-29T11:21:05.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.950+0000 7fb451196640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb44c145bd0 con 0x7fb44c13f650 2024-03-29T11:21:05.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.950+0000 7fb451196640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb44c145eb0 con 0x7fb44c1457f0 2024-03-29T11:21:05.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.950+0000 7fb451196640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:05.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.950+0000 7fb451196640 10 monclient(hunting): authenticate will time out at 890.268038s 2024-03-29T11:21:05.931 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.954+0000 7fb44a59c640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fb44c13f650 0x7fb44c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:55317/0 (socket says 172.21.15.86:55317) 2024-03-29T11:21:05.931 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.954+0000 7fb44a59c640 1 -- 172.21.15.86:0/3555985960 learned_addr learned my addr 172.21.15.86:0/3555985960 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:21:05.932 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.955+0000 7fb449d9b640 1 -- 172.21.15.86:0/3555985960 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 73741062 0 0) 0x7fb44c0aaef0 con 0x7fb44c0a9d70 2024-03-29T11:21:05.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.955+0000 7fb449d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:21:05.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.955+0000 7fb449d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:05.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.956+0000 7fb449d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:05.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.956+0000 7fb449d9b640 10 monclient(hunting): my global_id is 4251 2024-03-29T11:21:05.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.956+0000 7fb449d9b640 1 -- 172.21.15.86:0/3555985960 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb434003520 con 0x7fb44c0a9d70 2024-03-29T11:21:05.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.956+0000 7fb449d9b640 1 -- 172.21.15.86:0/3555985960 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1398734568 0 0) 0x7fb44c145bd0 con 0x7fb44c13f650 2024-03-29T11:21:05.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.956+0000 7fb449d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:21:05.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.956+0000 7fb449d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:05.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.956+0000 7fb449d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:05.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.956+0000 7fb449d9b640 10 monclient(hunting): my global_id is 4243 2024-03-29T11:21:05.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.956+0000 7fb449d9b640 1 -- 172.21.15.86:0/3555985960 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb44c0aaef0 con 0x7fb44c13f650 2024-03-29T11:21:05.936 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.959+0000 7fb449d9b640 1 -- 172.21.15.86:0/3555985960 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1568358500 0 0) 0x7fb434003520 con 0x7fb44c0a9d70 2024-03-29T11:21:05.937 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.959+0000 7fb449d9b640 1 -- 172.21.15.86:0/3555985960 >> v1:172.21.15.86:6790/0 conn(0x7fb44c1457f0 legacy=0x7fb44c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:05.937 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.959+0000 7fb449d9b640 1 -- 172.21.15.86:0/3555985960 >> v1:172.21.15.196:6789/0 conn(0x7fb44c13f650 legacy=0x7fb44c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:05.937 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.959+0000 7fb449d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:21:05.937 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.959+0000 7fb449d9b640 1 monclient: found mon.noname-a 2024-03-29T11:21:05.937 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.959+0000 7fb449d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:05.937 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.959+0000 7fb449d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.86:6789/0 2024-03-29T11:21:05.937 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.959+0000 7fb449d9b640 1 -- 172.21.15.86:0/3555985960 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb44c146340 con 0x7fb44c0a9d70 2024-03-29T11:21:05.937 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.959+0000 7fb449d9b640 10 monclient: _finish_auth 0 2024-03-29T11:21:05.937 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.959+0000 7fb449d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:21:05.937 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.959+0000 7fb449d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:05.937 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.959+0000 7fb449d9b640 1 -- 172.21.15.86:0/3555985960 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fb43c001690 con 0x7fb44c0a9d70 2024-03-29T11:21:05.937 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.959+0000 7fb449d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:05.937 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.959+0000 7fb449d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:05.938 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.959+0000 7fb449d9b640 10 monclient: dump: 2024-03-29T11:21:05.938 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:21:05.938 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:05.938 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:05.938 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:05.938 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:05.938 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:05.938 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:21:05.938 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:21:05.938 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:21:05.938 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:05.938 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.959+0000 7fb451196640 5 monclient: authenticate success, global_id 4251 2024-03-29T11:21:05.938 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.959+0000 7fb451196640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:05.938 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.959+0000 7fb449d9b640 1 -- 172.21.15.86:0/3555985960 <== mon.0 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb43c0017f0 con 0x7fb44c0a9d70 2024-03-29T11:21:05.939 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.959+0000 7fb449d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:05.939 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.959+0000 7fb449d9b640 1 -- 172.21.15.86:0/3555985960 <== mon.0 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fb43c001c40 con 0x7fb44c0a9d70 2024-03-29T11:21:05.939 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.959+0000 7fb449d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:05.939 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.959+0000 7fb449d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:05.939 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.959+0000 7fb449d9b640 10 monclient: dump: 2024-03-29T11:21:05.939 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:21:05.939 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:05.939 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:05.939 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:05.939 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:05.940 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:05.940 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:21:05.940 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:21:05.940 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:21:05.940 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:05.941 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.962+0000 7fb451196640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:05.941 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.962+0000 7fb451196640 4 set_mon_vals no callback set 2024-03-29T11:21:05.941 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.963+0000 7fb451196640 10 monclient: shutdown 2024-03-29T11:21:05.941 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.963+0000 7fb451196640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:05.941 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.963+0000 7fb451196640 1 -- 172.21.15.86:0/3555985960 >> v1:172.21.15.86:6789/0 conn(0x7fb44c0a9d70 legacy=0x7fb44c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:05.941 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.964+0000 7fb451196640 1 -- 172.21.15.86:0/3555985960 shutdown_connections 2024-03-29T11:21:05.941 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.964+0000 7fb451196640 1 -- 172.21.15.86:0/3555985960 >> 172.21.15.86:0/3555985960 conn(0x7fb44c066a00 msgr2=0x7fb44c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:05.942 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.964+0000 7fb451196640 1 -- 172.21.15.86:0/3555985960 shutdown_connections 2024-03-29T11:21:05.943 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.964+0000 7fb451196640 1 -- 172.21.15.86:0/3555985960 wait complete. 2024-03-29T11:21:05.943 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.966+0000 7fb451196640 10 monclient: build_initial_monmap 2024-03-29T11:21:05.943 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.966+0000 7fb451196640 10 monclient: monmap: 2024-03-29T11:21:05.943 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:21:05.943 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:05.943 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:21:05.943 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:21:05.943 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:05.943 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:05.943 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:21:05.943 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:21:05.944 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:21:05.944 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:05.944 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.966+0000 7fb451196640 1 librados: starting msgr at 2024-03-29T11:21:05.944 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.966+0000 7fb451196640 1 librados: starting objecter 2024-03-29T11:21:05.944 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.966+0000 7fb451196640 1 Processor -- start 2024-03-29T11:21:05.944 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.967+0000 7fb451196640 1 -- start start 2024-03-29T11:21:05.950 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.967+0000 7fb451196640 1 librados: setting wanted keys 2024-03-29T11:21:05.950 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.967+0000 7fb451196640 1 librados: calling monclient init 2024-03-29T11:21:05.950 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.967+0000 7fb451196640 10 monclient: init 2024-03-29T11:21:05.950 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.967+0000 7fb451196640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:05.950 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.967+0000 7fb451196640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:05.950 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.967+0000 7fb451196640 10 monclient(hunting): picked mon.noname-c con 0x7fb44c0a9d70 addr v1:172.21.15.86:6790/0 2024-03-29T11:21:05.951 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.967+0000 7fb451196640 10 monclient(hunting): picked mon.noname-b con 0x7fb44c13f650 addr v1:172.21.15.196:6789/0 2024-03-29T11:21:05.951 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.967+0000 7fb451196640 10 monclient(hunting): picked mon.noname-a con 0x7fb44c1457f0 addr v1:172.21.15.86:6789/0 2024-03-29T11:21:05.951 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.967+0000 7fb451196640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb44c072090 con 0x7fb44c1457f0 2024-03-29T11:21:05.951 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.967+0000 7fb451196640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb44c070b00 con 0x7fb44c13f650 2024-03-29T11:21:05.951 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.967+0000 7fb451196640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb44c070de0 con 0x7fb44c0a9d70 2024-03-29T11:21:05.951 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.967+0000 7fb451196640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:05.951 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.967+0000 7fb451196640 10 monclient(hunting): authenticate will time out at 890.285881s 2024-03-29T11:21:05.951 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.971+0000 7fb44a59c640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fb44c13f650 0x7fb44c06f730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:55429/0 (socket says 172.21.15.86:55429) 2024-03-29T11:21:05.951 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.971+0000 7fb44a59c640 1 -- 172.21.15.86:0/1487636461 learned_addr learned my addr 172.21.15.86:0/1487636461 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:21:05.952 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.971+0000 7fb4337fe640 10 client.?.objecter ms_handle_connect 0x7fb44c13f650 2024-03-29T11:21:05.952 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.972+0000 7fb4337fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:05.952 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.972+0000 7fb4337fe640 1 -- 172.21.15.86:0/1487636461 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3515151065 0 0) 0x7fb44c070b00 con 0x7fb44c13f650 2024-03-29T11:21:05.952 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.972+0000 7fb4337fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:05.952 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.972+0000 7fb4337fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:05.952 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.972+0000 7fb4337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:05.952 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.972+0000 7fb4337fe640 10 monclient(hunting): my global_id is 4246 2024-03-29T11:21:05.952 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.972+0000 7fb4337fe640 1 -- 172.21.15.86:0/1487636461 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb438004eb0 con 0x7fb44c13f650 2024-03-29T11:21:05.952 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.974+0000 7fb4337fe640 10 client.?.objecter ms_handle_connect 0x7fb44c0a9d70 2024-03-29T11:21:05.959 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.974+0000 7fb4337fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:05.959 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.974+0000 7fb4337fe640 1 -- 172.21.15.86:0/1487636461 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4137498911 0 0) 0x7fb44c070de0 con 0x7fb44c0a9d70 2024-03-29T11:21:05.959 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.974+0000 7fb4337fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:05.960 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.974+0000 7fb4337fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:05.960 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.974+0000 7fb4337fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:05.960 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.974+0000 7fb4337fe640 10 monclient(hunting): my global_id is 4247 2024-03-29T11:21:05.960 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.974+0000 7fb4337fe640 1 -- 172.21.15.86:0/1487636461 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb44c070b00 con 0x7fb44c0a9d70 2024-03-29T11:21:05.960 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.975+0000 7fb4337fe640 1 -- 172.21.15.86:0/1487636461 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1831901659 0 0) 0x7fb44c070b00 con 0x7fb44c0a9d70 2024-03-29T11:21:05.960 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.975+0000 7fb4337fe640 1 -- 172.21.15.86:0/1487636461 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb44c070de0 con 0x7fb44c0a9d70 2024-03-29T11:21:05.960 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.975+0000 7fb4337fe640 1 -- 172.21.15.86:0/1487636461 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fb440001790 con 0x7fb44c0a9d70 2024-03-29T11:21:05.960 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.975+0000 7fb4337fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:05.960 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.976+0000 7fb4337fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:05.960 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.976+0000 7fb4337fe640 10 monclient(hunting): dump: 2024-03-29T11:21:05.961 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:21:05.961 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:05.961 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:05.961 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:05.961 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:05.961 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:05.961 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:21:05.961 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:21:05.961 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:21:05.961 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:05.961 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.976+0000 7fb4337fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:05.961 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.976+0000 7fb4337fe640 1 -- 172.21.15.86:0/1487636461 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3833429414 0 0) 0x7fb44c070de0 con 0x7fb44c0a9d70 2024-03-29T11:21:05.961 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.976+0000 7fb4337fe640 1 -- 172.21.15.86:0/1487636461 >> v1:172.21.15.196:6789/0 conn(0x7fb44c13f650 legacy=0x7fb44c06f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:05.961 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.976+0000 7fb4337fe640 1 -- 172.21.15.86:0/1487636461 >> v1:172.21.15.86:6789/0 conn(0x7fb44c1457f0 legacy=0x7fb44c1e5e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:05.962 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.976+0000 7fb4337fe640 10 monclient: _finish_hunting 0 2024-03-29T11:21:05.962 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.976+0000 7fb4337fe640 1 monclient: found mon.c 2024-03-29T11:21:05.962 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.976+0000 7fb4337fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:05.962 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.976+0000 7fb4337fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:21:05.962 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.976+0000 7fb4337fe640 1 -- 172.21.15.86:0/1487636461 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb44c0710c0 con 0x7fb44c0a9d70 2024-03-29T11:21:05.962 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.976+0000 7fb4337fe640 10 monclient: _finish_auth 0 2024-03-29T11:21:05.962 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.976+0000 7fb4337fe640 10 monclient: _check_auth_tickets 2024-03-29T11:21:05.962 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.976+0000 7fb4337fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:05.962 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.976+0000 7fb451196640 5 monclient: authenticate success, global_id 4247 2024-03-29T11:21:05.963 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.976+0000 7fb451196640 10 monclient: _renew_subs 2024-03-29T11:21:05.963 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.976+0000 7fb451196640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:21:05.963 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.976+0000 7fb451196640 1 -- 172.21.15.86:0/1487636461 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb44c1427b0 con 0x7fb44c0a9d70 2024-03-29T11:21:05.963 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.977+0000 7fb451196640 10 client.4247.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:05.963 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.977+0000 7fb451196640 10 monclient: _renew_subs 2024-03-29T11:21:05.963 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.977+0000 7fb451196640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:21:05.963 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.977+0000 7fb451196640 1 -- 172.21.15.86:0/1487636461 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb44c131ce0 con 0x7fb44c0a9d70 2024-03-29T11:21:05.963 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.977+0000 7fb451196640 1 librados: init done 2024-03-29T11:21:05.963 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.978+0000 7fb4337fe640 1 -- 172.21.15.86:0/1487636461 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb440001ca0 con 0x7fb44c0a9d70 2024-03-29T11:21:05.963 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.978+0000 7fb4337fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:05.963 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.978+0000 7fb4337fe640 1 -- 172.21.15.86:0/1487636461 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fb440003790 con 0x7fb44c0a9d70 2024-03-29T11:21:05.963 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.978+0000 7fb4337fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:05.964 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.978+0000 7fb4337fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:05.964 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.979+0000 7fb451196640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:05.964 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.979+0000 7fb4337fe640 10 monclient: dump: 2024-03-29T11:21:05.964 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:21:05.964 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:05.964 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:05.964 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:05.964 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:05.964 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:05.964 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:21:05.964 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:21:05.964 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:21:05.964 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:05.964 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.979+0000 7fb451196640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:05.964 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.979+0000 7fb451196640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:21:05.965 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.979+0000 7fb451196640 1 -- 172.21.15.86:0/1487636461 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb44c131f90 con 0x7fb44c0a9d70 2024-03-29T11:21:05.965 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.979+0000 7fb44959a640 4 set_mon_vals no callback set 2024-03-29T11:21:05.965 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.980+0000 7fb4337fe640 1 -- 172.21.15.86:0/1487636461 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7fb440019760 con 0x7fb44c0a9d70 2024-03-29T11:21:05.965 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.980+0000 7fb44a59c640 10 monclient: get_auth_request con 0x7fb43806dce0 auth_method 0 2024-03-29T11:21:05.965 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.980+0000 7fb4337fe640 1 -- 172.21.15.86:0/1487636461 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7fb4400837a0 con 0x7fb44c0a9d70 2024-03-29T11:21:05.965 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.980+0000 7fb4337fe640 10 client.4247.objecter ms_dispatch 0x7fb44c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:05.965 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.980+0000 7fb4337fe640 3 client.4247.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:05.965 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.980+0000 7fb4337fe640 3 client.4247.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:05.965 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.980+0000 7fb4337fe640 20 client.4247.objecter dump_active .. 0 homeless 2024-03-29T11:21:05.965 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.981+0000 7fb4337fe640 10 client.4247.objecter ms_handle_connect 0x7fb43806dce0 2024-03-29T11:21:05.967 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.991+0000 7fb4337fe640 1 -- 172.21.15.86:0/1487636461 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb44c131f90 con 0x7fb44c0a9d70 2024-03-29T11:21:05.967 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.991+0000 7fb4337fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:05.967 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:05.991+0000 7fb4337fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:06.293 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:06.316+0000 7fb451196640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:06.293 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:06.316+0000 7fb451196640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:06.293 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:06.316+0000 7fb451196640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:21:06.293 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:06.316+0000 7fb451196640 1 -- 172.21.15.86:0/1487636461 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb44c131f90 con 0x7fb44c0a9d70 2024-03-29T11:21:06.294 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:06.318+0000 7fb4337fe640 1 -- 172.21.15.86:0/1487636461 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7fb44c131f90 con 0x7fb44c0a9d70 2024-03-29T11:21:06.294 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:06.318+0000 7fb4337fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:06.294 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:06.318+0000 7fb4337fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:06.298 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:21:06.298 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:06.299 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:06.322+0000 7fb451196640 10 librados: watch_flush enter 2024-03-29T11:21:06.299 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:06.322+0000 7fb451196640 10 librados: watch_flush exit 2024-03-29T11:21:06.299 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:06.322+0000 7fb451196640 20 client.4247.objecter shutdown clearing up homeless session... 2024-03-29T11:21:06.299 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:06.322+0000 7fb451196640 10 client.4247.objecter successfully canceled tick 2024-03-29T11:21:06.299 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:06.323+0000 7fb451196640 1 -- 172.21.15.86:0/1487636461 >> v1:172.21.15.196:6816/588110318 conn(0x7fb43806dce0 legacy=0x7fb4380701a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:06.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:06.323+0000 7fb451196640 10 monclient: shutdown 2024-03-29T11:21:06.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:06.323+0000 7fb451196640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:06.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:06.323+0000 7fb451196640 1 -- 172.21.15.86:0/1487636461 >> v1:172.21.15.86:6790/0 conn(0x7fb44c0a9d70 legacy=0x7fb44c071850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:06.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:06.323+0000 7fb451196640 1 -- 172.21.15.86:0/1487636461 shutdown_connections 2024-03-29T11:21:06.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:06.323+0000 7fb451196640 1 -- 172.21.15.86:0/1487636461 >> 172.21.15.86:0/1487636461 conn(0x7fb44c066a00 msgr2=0x7fb44c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:06.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:06.323+0000 7fb451196640 1 -- 172.21.15.86:0/1487636461 shutdown_connections 2024-03-29T11:21:06.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:06.323+0000 7fb451196640 1 -- 172.21.15.86:0/1487636461 wait complete. 2024-03-29T11:21:06.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:06.323+0000 7fb451196640 1 librados: shutdown 2024-03-29T11:21:06.318 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:13.125 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:13.283 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.306+0000 7f4243eb3640 10 monclient: get_monmap_and_config 2024-03-29T11:21:13.283 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.306+0000 7f4243eb3640 10 monclient: build_initial_monmap 2024-03-29T11:21:13.283 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.306+0000 7f4243eb3640 10 monclient: monmap: 2024-03-29T11:21:13.283 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:21:13.283 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:13.283 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:21:13.307078+0000 2024-03-29T11:21:13.283 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:21:13.307078+0000 2024-03-29T11:21:13.283 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:13.283 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:13.283 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:21:13.284 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:21:13.284 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:21:13.284 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:13.285 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.308+0000 7f4243eb3640 1 Processor -- start 2024-03-29T11:21:13.285 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.308+0000 7f4243eb3640 1 -- start start 2024-03-29T11:21:13.285 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.308+0000 7f4243eb3640 10 monclient: init 2024-03-29T11:21:13.286 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.309+0000 7f4243eb3640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:13.286 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.309+0000 7f4243eb3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:21:13.286 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.309+0000 7f4243eb3640 10 monclient(hunting): picked mon.noname-b con 0x7f423c0a9ac0 addr v1:172.21.15.86:6790/0 2024-03-29T11:21:13.286 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.309+0000 7f4243eb3640 10 monclient(hunting): picked mon.noname-c con 0x7f423c13ee90 addr v1:172.21.15.196:6789/0 2024-03-29T11:21:13.286 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.309+0000 7f4243eb3640 10 monclient(hunting): picked mon.noname-a con 0x7f423c0af900 addr v1:172.21.15.86:6789/0 2024-03-29T11:21:13.286 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.309+0000 7f4243eb3640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f423c0afd10 con 0x7f423c0af900 2024-03-29T11:21:13.286 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.309+0000 7f4243eb3640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f423c0af6f0 con 0x7f423c13ee90 2024-03-29T11:21:13.286 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.309+0000 7f4243eb3640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f423c0aac80 con 0x7f423c0a9ac0 2024-03-29T11:21:13.286 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.309+0000 7f4243eb3640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:13.286 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.309+0000 7f4243eb3640 10 monclient(hunting): authenticate will time out at 897.627432s 2024-03-29T11:21:13.288 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.312+0000 7f4241c4f640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f423c0a9ac0 0x7f423c0adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:57715/0 (socket says 172.21.15.86:57715) 2024-03-29T11:21:13.289 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.312+0000 7f4241c4f640 1 -- 172.21.15.86:0/2914145284 learned_addr learned my addr 172.21.15.86:0/2914145284 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:21:13.290 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.312+0000 7f4240c4d640 1 -- 172.21.15.86:0/2914145284 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 741023800 0 0) 0x7f423c0aac80 con 0x7f423c0a9ac0 2024-03-29T11:21:13.290 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.312+0000 7f4240c4d640 10 monclient(hunting): _negotiate 2024-03-29T11:21:13.290 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.312+0000 7f4240c4d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:13.290 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.312+0000 7f4240c4d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:13.291 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.312+0000 7f4240c4d640 10 monclient(hunting): my global_id is 4250 2024-03-29T11:21:13.291 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.312+0000 7f4240c4d640 1 -- 172.21.15.86:0/2914145284 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4228003520 con 0x7f423c0a9ac0 2024-03-29T11:21:13.291 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.313+0000 7f4240c4d640 1 -- 172.21.15.86:0/2914145284 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1592701939 0 0) 0x7f423c0afd10 con 0x7f423c0af900 2024-03-29T11:21:13.291 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.313+0000 7f4240c4d640 10 monclient(hunting): _negotiate 2024-03-29T11:21:13.291 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.313+0000 7f4240c4d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:13.291 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.313+0000 7f4240c4d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:13.291 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.313+0000 7f4240c4d640 10 monclient(hunting): my global_id is 4254 2024-03-29T11:21:13.291 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.313+0000 7f4240c4d640 1 -- 172.21.15.86:0/2914145284 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f423c0aac80 con 0x7f423c0af900 2024-03-29T11:21:13.291 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.313+0000 7f4240c4d640 1 -- 172.21.15.86:0/2914145284 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 387091495 0 0) 0x7f423c0aac80 con 0x7f423c0af900 2024-03-29T11:21:13.291 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.313+0000 7f4240c4d640 1 -- 172.21.15.86:0/2914145284 >> v1:172.21.15.86:6790/0 conn(0x7f423c0a9ac0 legacy=0x7f423c0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:13.291 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.313+0000 7f4240c4d640 1 -- 172.21.15.86:0/2914145284 >> v1:172.21.15.196:6789/0 conn(0x7f423c13ee90 legacy=0x7f423c13b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:13.294 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.314+0000 7f4240c4d640 10 monclient: _finish_hunting 0 2024-03-29T11:21:13.294 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.314+0000 7f4240c4d640 1 monclient: found mon.noname-a 2024-03-29T11:21:13.294 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.315+0000 7f4240c4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:13.294 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.315+0000 7f4240c4d640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.86:6789/0 2024-03-29T11:21:13.294 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.315+0000 7f4240c4d640 1 -- 172.21.15.86:0/2914145284 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f423c0aaf60 con 0x7f423c0af900 2024-03-29T11:21:13.294 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.315+0000 7f4240c4d640 10 monclient: _finish_auth 0 2024-03-29T11:21:13.294 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.315+0000 7f4240c4d640 10 monclient: _check_auth_tickets 2024-03-29T11:21:13.294 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.315+0000 7f4240c4d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:13.294 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.315+0000 7f4240c4d640 1 -- 172.21.15.86:0/2914145284 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f42340019f0 con 0x7f423c0af900 2024-03-29T11:21:13.295 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.315+0000 7f4243eb3640 5 monclient: authenticate success, global_id 4254 2024-03-29T11:21:13.295 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.315+0000 7f4243eb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:13.295 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.315+0000 7f4240c4d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:13.295 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.315+0000 7f4240c4d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:13.295 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.315+0000 7f4240c4d640 10 monclient: dump: 2024-03-29T11:21:13.295 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:21:13.295 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:13.295 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:13.295 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:13.295 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:13.295 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:13.295 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:21:13.295 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:21:13.295 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:21:13.295 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:13.296 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.315+0000 7f4243eb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:13.296 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.315+0000 7f4240c4d640 1 -- 172.21.15.86:0/2914145284 <== mon.0 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f4234001b50 con 0x7f423c0af900 2024-03-29T11:21:13.296 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.316+0000 7f4240c4d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:13.296 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.316+0000 7f4240c4d640 1 -- 172.21.15.86:0/2914145284 <== mon.0 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f4234001fa0 con 0x7f423c0af900 2024-03-29T11:21:13.296 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.316+0000 7f4243eb3640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:13.296 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.316+0000 7f4243eb3640 4 set_mon_vals no callback set 2024-03-29T11:21:13.296 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.317+0000 7f4243eb3640 10 monclient: shutdown 2024-03-29T11:21:13.296 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.317+0000 7f4243eb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:13.296 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.317+0000 7f4243eb3640 1 -- 172.21.15.86:0/2914145284 >> v1:172.21.15.86:6789/0 conn(0x7f423c0af900 legacy=0x7f423c149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:13.296 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.317+0000 7f4240c4d640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:21:13.297 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.319+0000 7f4243eb3640 1 -- 172.21.15.86:0/2914145284 shutdown_connections 2024-03-29T11:21:13.297 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.319+0000 7f4243eb3640 1 -- 172.21.15.86:0/2914145284 >> 172.21.15.86:0/2914145284 conn(0x7f423c0b35e0 msgr2=0x7f423c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:13.297 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.320+0000 7f4243eb3640 1 -- 172.21.15.86:0/2914145284 shutdown_connections 2024-03-29T11:21:13.297 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.320+0000 7f4243eb3640 1 -- 172.21.15.86:0/2914145284 wait complete. 2024-03-29T11:21:13.298 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.320+0000 7f4243eb3640 10 monclient: build_initial_monmap 2024-03-29T11:21:13.298 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.320+0000 7f4243eb3640 10 monclient: monmap: 2024-03-29T11:21:13.298 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:21:13.298 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:13.298 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:21:13.298 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:21:13.298 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:13.298 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:13.298 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:21:13.298 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:21:13.298 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:21:13.299 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:13.299 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.321+0000 7f4243eb3640 1 librados: starting msgr at 2024-03-29T11:21:13.299 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.321+0000 7f4243eb3640 1 librados: starting objecter 2024-03-29T11:21:13.299 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.321+0000 7f4243eb3640 1 Processor -- start 2024-03-29T11:21:13.301 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.324+0000 7f4243eb3640 1 -- start start 2024-03-29T11:21:13.301 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.324+0000 7f4243eb3640 1 librados: setting wanted keys 2024-03-29T11:21:13.301 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.324+0000 7f4243eb3640 1 librados: calling monclient init 2024-03-29T11:21:13.301 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.324+0000 7f4243eb3640 10 monclient: init 2024-03-29T11:21:13.302 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.325+0000 7f4243eb3640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:13.302 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.325+0000 7f4243eb3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:21:13.306 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.325+0000 7f4243eb3640 10 monclient(hunting): picked mon.noname-a con 0x7f423c0a9ac0 addr v1:172.21.15.86:6789/0 2024-03-29T11:21:13.306 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.325+0000 7f4243eb3640 10 monclient(hunting): picked mon.noname-c con 0x7f423c0af900 addr v1:172.21.15.86:6790/0 2024-03-29T11:21:13.306 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.325+0000 7f4243eb3640 10 monclient(hunting): picked mon.noname-b con 0x7f423c13ee90 addr v1:172.21.15.196:6789/0 2024-03-29T11:21:13.306 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.325+0000 7f4243eb3640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f423c070160 con 0x7f423c0a9ac0 2024-03-29T11:21:13.306 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.325+0000 7f4243eb3640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f423c1422b0 con 0x7f423c13ee90 2024-03-29T11:21:13.306 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.325+0000 7f4243eb3640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f423c070440 con 0x7f423c0af900 2024-03-29T11:21:13.306 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.325+0000 7f4243eb3640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:13.306 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.325+0000 7f4243eb3640 10 monclient(hunting): authenticate will time out at 897.643778s 2024-03-29T11:21:13.306 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.329+0000 7f424144e640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f423c0af900 0x7f423c13f7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:57829/0 (socket says 172.21.15.86:57829) 2024-03-29T11:21:13.306 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.329+0000 7f4241c4f640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f423c0a9ac0 0x7f423c070e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:54013/0 (socket says 172.21.15.86:54013) 2024-03-29T11:21:13.307 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.329+0000 7f4241c4f640 1 -- 172.21.15.86:0/2518338494 learned_addr learned my addr 172.21.15.86:0/2518338494 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:21:13.307 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.329+0000 7f42327fc640 10 client.?.objecter ms_handle_connect 0x7f423c0a9ac0 2024-03-29T11:21:13.307 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.330+0000 7f42327fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:13.311 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.330+0000 7f42327fc640 10 client.?.objecter ms_handle_connect 0x7f423c0af900 2024-03-29T11:21:13.311 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.330+0000 7f42327fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:13.311 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.330+0000 7f42327fc640 1 -- 172.21.15.86:0/2518338494 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3205474391 0 0) 0x7f423c070160 con 0x7f423c0a9ac0 2024-03-29T11:21:13.311 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.330+0000 7f42327fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:13.311 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.330+0000 7f42327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:13.311 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.330+0000 7f42327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:13.311 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.330+0000 7f42327fc640 10 monclient(hunting): my global_id is 4257 2024-03-29T11:21:13.312 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.330+0000 7f42327fc640 1 -- 172.21.15.86:0/2518338494 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4220004eb0 con 0x7f423c0a9ac0 2024-03-29T11:21:13.312 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.331+0000 7f42327fc640 1 -- 172.21.15.86:0/2518338494 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4013668544 0 0) 0x7f423c070440 con 0x7f423c0af900 2024-03-29T11:21:13.312 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.331+0000 7f42327fc640 10 monclient(hunting): _negotiate 2024-03-29T11:21:13.312 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.331+0000 7f42327fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:13.312 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.331+0000 7f42327fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:13.312 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.331+0000 7f42327fc640 10 monclient(hunting): my global_id is 4253 2024-03-29T11:21:13.312 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.331+0000 7f42327fc640 1 -- 172.21.15.86:0/2518338494 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f423c070160 con 0x7f423c0af900 2024-03-29T11:21:13.312 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.332+0000 7f42327fc640 1 -- 172.21.15.86:0/2518338494 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2330027230 0 0) 0x7f4220004eb0 con 0x7f423c0a9ac0 2024-03-29T11:21:13.312 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.332+0000 7f42327fc640 1 -- 172.21.15.86:0/2518338494 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f423c070440 con 0x7f423c0a9ac0 2024-03-29T11:21:13.312 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.332+0000 7f42327fc640 1 -- 172.21.15.86:0/2518338494 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f422c001990 con 0x7f423c0a9ac0 2024-03-29T11:21:13.312 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.332+0000 7f42327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:13.312 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.332+0000 7f42327fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:13.312 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.332+0000 7f42327fc640 10 monclient(hunting): dump: 2024-03-29T11:21:13.312 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:21:13.313 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:13.313 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:13.313 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:13.313 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:13.313 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:13.313 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:21:13.313 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:21:13.313 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:21:13.313 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:13.313 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.332+0000 7f42327fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:13.313 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.332+0000 7f42327fc640 1 -- 172.21.15.86:0/2518338494 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3064543327 0 0) 0x7f423c070160 con 0x7f423c0af900 2024-03-29T11:21:13.313 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.332+0000 7f42327fc640 1 -- 172.21.15.86:0/2518338494 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4220004eb0 con 0x7f423c0af900 2024-03-29T11:21:13.313 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.332+0000 7f42327fc640 1 -- 172.21.15.86:0/2518338494 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f4238001790 con 0x7f423c0af900 2024-03-29T11:21:13.314 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.332+0000 7f42327fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:13.314 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.332+0000 7f42327fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:13.314 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.332+0000 7f42327fc640 10 monclient(hunting): dump: 2024-03-29T11:21:13.314 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:21:13.314 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:13.314 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:13.314 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:13.314 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:13.314 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:13.314 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:21:13.314 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:21:13.314 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:21:13.314 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:13.314 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.332+0000 7f42327fc640 1 -- 172.21.15.86:0/2518338494 <== mon.0 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1549938497 0 0) 0x7f423c070440 con 0x7f423c0a9ac0 2024-03-29T11:21:13.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.332+0000 7f42327fc640 1 -- 172.21.15.86:0/2518338494 >> v1:172.21.15.86:6790/0 conn(0x7f423c0af900 legacy=0x7f423c13f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:13.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.332+0000 7f42327fc640 1 -- 172.21.15.86:0/2518338494 >> v1:172.21.15.196:6789/0 conn(0x7f423c13ee90 legacy=0x7f423c1e5fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:13.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.332+0000 7f42327fc640 10 monclient: _finish_hunting 0 2024-03-29T11:21:13.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.332+0000 7f42327fc640 1 monclient: found mon.a 2024-03-29T11:21:13.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.332+0000 7f42327fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:13.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.332+0000 7f42327fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:21:13.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.332+0000 7f42327fc640 1 -- 172.21.15.86:0/2518338494 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f423c070720 con 0x7f423c0a9ac0 2024-03-29T11:21:13.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.332+0000 7f42327fc640 10 monclient: _finish_auth 0 2024-03-29T11:21:13.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.332+0000 7f42327fc640 10 monclient: _check_auth_tickets 2024-03-29T11:21:13.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.332+0000 7f42327fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:13.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.332+0000 7f4243eb3640 5 monclient: authenticate success, global_id 4257 2024-03-29T11:21:13.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.332+0000 7f4243eb3640 10 monclient: _renew_subs 2024-03-29T11:21:13.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.332+0000 7f4243eb3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:21:13.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.332+0000 7f4243eb3640 1 -- 172.21.15.86:0/2518338494 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f423c0715a0 con 0x7f423c0a9ac0 2024-03-29T11:21:13.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.333+0000 7f4243eb3640 10 client.4257.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:13.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.333+0000 7f4243eb3640 10 monclient: _renew_subs 2024-03-29T11:21:13.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.333+0000 7f4243eb3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:21:13.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.333+0000 7f4243eb3640 1 -- 172.21.15.86:0/2518338494 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f423c06e9f0 con 0x7f423c0a9ac0 2024-03-29T11:21:13.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.333+0000 7f4243eb3640 1 librados: init done 2024-03-29T11:21:13.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.333+0000 7f42327fc640 1 -- 172.21.15.86:0/2518338494 <== mon.0 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f422c0047e0 con 0x7f423c0a9ac0 2024-03-29T11:21:13.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.333+0000 7f42327fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:13.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.333+0000 7f4233fff640 4 set_mon_vals no callback set 2024-03-29T11:21:13.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.334+0000 7f42327fc640 1 -- 172.21.15.86:0/2518338494 <== mon.0 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f422c004af0 con 0x7f423c0a9ac0 2024-03-29T11:21:13.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.334+0000 7f42327fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:13.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.334+0000 7f42327fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:13.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.334+0000 7f42327fc640 10 monclient: dump: 2024-03-29T11:21:13.316 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:21:13.316 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:13.316 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:13.316 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:13.317 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:13.317 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:13.317 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:21:13.317 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:21:13.317 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:21:13.317 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:13.317 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.334+0000 7f4243eb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:13.317 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.334+0000 7f4243eb3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:13.317 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.334+0000 7f4243eb3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:21:13.317 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.334+0000 7f4243eb3640 1 -- 172.21.15.86:0/2518338494 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4224005080 con 0x7f423c0a9ac0 2024-03-29T11:21:13.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.336+0000 7f42327fc640 1 -- 172.21.15.86:0/2518338494 <== mon.0 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f422c01aac0 con 0x7f423c0a9ac0 2024-03-29T11:21:13.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.337+0000 7f42327fc640 1 -- 172.21.15.86:0/2518338494 <== mon.0 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f422c084ae0 con 0x7f423c0a9ac0 2024-03-29T11:21:13.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.337+0000 7f424144e640 10 monclient: get_auth_request con 0x7f422006de80 auth_method 0 2024-03-29T11:21:13.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.338+0000 7f42327fc640 10 client.4257.objecter ms_dispatch 0x7f423c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:13.319 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.338+0000 7f42327fc640 3 client.4257.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:13.319 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.338+0000 7f42327fc640 3 client.4257.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:13.319 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.338+0000 7f42327fc640 20 client.4257.objecter dump_active .. 0 homeless 2024-03-29T11:21:13.319 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.338+0000 7f42327fc640 10 client.4257.objecter ms_handle_connect 0x7f422006de80 2024-03-29T11:21:13.324 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.348+0000 7f42327fc640 1 -- 172.21.15.86:0/2518338494 <== mon.0 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f422c0501f0 con 0x7f423c0a9ac0 2024-03-29T11:21:13.324 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.348+0000 7f42327fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:13.324 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.348+0000 7f42327fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:13.649 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.673+0000 7f4243eb3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:13.649 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.673+0000 7f4243eb3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:13.650 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.673+0000 7f4243eb3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:21:13.650 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.673+0000 7f4243eb3640 1 -- 172.21.15.86:0/2518338494 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4224005330 con 0x7f423c0a9ac0 2024-03-29T11:21:13.652 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.676+0000 7f42327fc640 1 -- 172.21.15.86:0/2518338494 <== mon.0 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f422c0522f0 con 0x7f423c0a9ac0 2024-03-29T11:21:13.653 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:21:13.653 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:13.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.676+0000 7f42327fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:13.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.676+0000 7f42327fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:13.657 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.680+0000 7f4243eb3640 10 librados: watch_flush enter 2024-03-29T11:21:13.659 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.681+0000 7f4243eb3640 10 librados: watch_flush exit 2024-03-29T11:21:13.659 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.681+0000 7f4243eb3640 20 client.4257.objecter shutdown clearing up homeless session... 2024-03-29T11:21:13.659 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.681+0000 7f4243eb3640 10 client.4257.objecter successfully canceled tick 2024-03-29T11:21:13.659 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.682+0000 7f4243eb3640 1 -- 172.21.15.86:0/2518338494 >> v1:172.21.15.196:6816/588110318 conn(0x7f422006de80 legacy=0x7f4220070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:13.659 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.682+0000 7f4243eb3640 10 monclient: shutdown 2024-03-29T11:21:13.659 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.682+0000 7f4243eb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:13.659 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.682+0000 7f4243eb3640 1 -- 172.21.15.86:0/2518338494 >> v1:172.21.15.86:6789/0 conn(0x7f423c0a9ac0 legacy=0x7f423c070e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:13.659 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.682+0000 7f4243eb3640 1 -- 172.21.15.86:0/2518338494 shutdown_connections 2024-03-29T11:21:13.659 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.682+0000 7f4243eb3640 1 -- 172.21.15.86:0/2518338494 >> 172.21.15.86:0/2518338494 conn(0x7f423c0b35e0 msgr2=0x7f423c13b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:13.664 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.687+0000 7f4243eb3640 1 -- 172.21.15.86:0/2518338494 shutdown_connections 2024-03-29T11:21:13.664 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.688+0000 7f4243eb3640 1 -- 172.21.15.86:0/2518338494 wait complete. 2024-03-29T11:21:13.664 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:13.688+0000 7f4243eb3640 1 librados: shutdown 2024-03-29T11:21:13.681 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:20.488 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:20.637 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.660+0000 7f0ead34d640 10 monclient: get_monmap_and_config 2024-03-29T11:21:20.637 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.660+0000 7f0ead34d640 10 monclient: build_initial_monmap 2024-03-29T11:21:20.637 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.660+0000 7f0ead34d640 10 monclient: monmap: 2024-03-29T11:21:20.637 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:21:20.637 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:20.637 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:21:20.661376+0000 2024-03-29T11:21:20.637 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:21:20.661376+0000 2024-03-29T11:21:20.637 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:20.638 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:20.638 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:21:20.638 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:21:20.638 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:21:20.638 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:20.639 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.662+0000 7f0ead34d640 1 Processor -- start 2024-03-29T11:21:20.639 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.662+0000 7f0ead34d640 1 -- start start 2024-03-29T11:21:20.639 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.662+0000 7f0ead34d640 10 monclient: init 2024-03-29T11:21:20.641 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.663+0000 7f0ead34d640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:20.641 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.664+0000 7f0ead34d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:21:20.641 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.664+0000 7f0ead34d640 10 monclient(hunting): picked mon.noname-c con 0x7f0ea80a9ac0 addr v1:172.21.15.196:6789/0 2024-03-29T11:21:20.641 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.664+0000 7f0ead34d640 10 monclient(hunting): picked mon.noname-a con 0x7f0ea813ee90 addr v1:172.21.15.86:6789/0 2024-03-29T11:21:20.641 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.664+0000 7f0ead34d640 10 monclient(hunting): picked mon.noname-b con 0x7f0ea80af900 addr v1:172.21.15.86:6790/0 2024-03-29T11:21:20.641 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.664+0000 7f0ead34d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ea80afd10 con 0x7f0ea813ee90 2024-03-29T11:21:20.641 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.664+0000 7f0ead34d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ea80af6f0 con 0x7f0ea80a9ac0 2024-03-29T11:21:20.641 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.664+0000 7f0ead34d640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ea80aac80 con 0x7f0ea80af900 2024-03-29T11:21:20.641 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.664+0000 7f0ead34d640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:20.641 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.664+0000 7f0ead34d640 10 monclient(hunting): authenticate will time out at 904.982607s 2024-03-29T11:21:20.645 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.669+0000 7f0ea77fe640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f0ea80af900 0x7f0ea8149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:57679/0 (socket says 172.21.15.86:57679) 2024-03-29T11:21:20.645 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.669+0000 7f0ea77fe640 1 -- 172.21.15.86:0/1202684648 learned_addr learned my addr 172.21.15.86:0/1202684648 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:21:20.646 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.670+0000 7f0ea5ffb640 1 -- 172.21.15.86:0/1202684648 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4076285289 0 0) 0x7f0ea80aac80 con 0x7f0ea80af900 2024-03-29T11:21:20.646 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.670+0000 7f0ea5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:20.646 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.670+0000 7f0ea5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:20.646 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.670+0000 7f0ea5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:20.646 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.670+0000 7f0ea5ffb640 10 monclient(hunting): my global_id is 4256 2024-03-29T11:21:20.646 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.670+0000 7f0ea5ffb640 1 -- 172.21.15.86:0/1202684648 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e90003520 con 0x7f0ea80af900 2024-03-29T11:21:20.647 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.670+0000 7f0ea5ffb640 1 -- 172.21.15.86:0/1202684648 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3786052770 0 0) 0x7f0e90003520 con 0x7f0ea80af900 2024-03-29T11:21:20.647 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.670+0000 7f0ea5ffb640 1 -- 172.21.15.86:0/1202684648 >> v1:172.21.15.196:6789/0 conn(0x7f0ea80a9ac0 legacy=0x7f0ea80adfe0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:21:20.647 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.670+0000 7f0ea5ffb640 1 -- 172.21.15.86:0/1202684648 >> v1:172.21.15.86:6789/0 conn(0x7f0ea813ee90 legacy=0x7f0ea813b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:20.649 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.673+0000 7f0ea5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:21:20.649 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.673+0000 7f0ea5ffb640 1 monclient: found mon.noname-b 2024-03-29T11:21:20.650 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.673+0000 7f0ea5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:20.650 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.673+0000 7f0ea5ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:21:20.650 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.673+0000 7f0ea5ffb640 1 -- 172.21.15.86:0/1202684648 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea80aaf60 con 0x7f0ea80af900 2024-03-29T11:21:20.650 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.673+0000 7f0ea5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:21:20.650 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.673+0000 7f0ea5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:21:20.650 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.673+0000 7f0ea5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:20.650 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.673+0000 7f0ead34d640 5 monclient: authenticate success, global_id 4256 2024-03-29T11:21:20.650 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.673+0000 7f0ead34d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:20.650 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.673+0000 7f0ea5ffb640 1 -- 172.21.15.86:0/1202684648 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f0ea0001a10 con 0x7f0ea80af900 2024-03-29T11:21:20.650 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.673+0000 7f0ea5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:20.650 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.673+0000 7f0ea5ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:20.650 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.673+0000 7f0ea5ffb640 10 monclient: dump: 2024-03-29T11:21:20.650 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:21:20.650 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:20.651 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:20.651 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:20.651 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:20.651 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:20.651 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:21:20.651 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:21:20.651 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:21:20.651 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:20.651 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.673+0000 7f0ead34d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:20.652 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.675+0000 7f0ea5ffb640 1 -- 172.21.15.86:0/1202684648 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0ea0001c70 con 0x7f0ea80af900 2024-03-29T11:21:20.652 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.675+0000 7f0ea5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:20.652 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.675+0000 7f0ea5ffb640 1 -- 172.21.15.86:0/1202684648 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f0ea00020c0 con 0x7f0ea80af900 2024-03-29T11:21:20.652 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.675+0000 7f0ea5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:20.652 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.675+0000 7f0ea5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:20.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.675+0000 7f0ea5ffb640 10 monclient: dump: 2024-03-29T11:21:20.653 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:21:20.653 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:20.653 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:20.653 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:20.653 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:20.653 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:20.653 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:21:20.653 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:21:20.653 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:21:20.653 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:20.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.675+0000 7f0ead34d640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:20.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.675+0000 7f0ead34d640 4 set_mon_vals no callback set 2024-03-29T11:21:20.654 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.676+0000 7f0ead34d640 10 monclient: shutdown 2024-03-29T11:21:20.654 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.677+0000 7f0ead34d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:20.654 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.677+0000 7f0ead34d640 1 -- 172.21.15.86:0/1202684648 >> v1:172.21.15.86:6790/0 conn(0x7f0ea80af900 legacy=0x7f0ea8149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:20.654 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.677+0000 7f0ead34d640 1 -- 172.21.15.86:0/1202684648 shutdown_connections 2024-03-29T11:21:20.654 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.677+0000 7f0ead34d640 1 -- 172.21.15.86:0/1202684648 >> 172.21.15.86:0/1202684648 conn(0x7f0ea80b35e0 msgr2=0x7f0ea80b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:20.659 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.683+0000 7f0ead34d640 1 -- 172.21.15.86:0/1202684648 shutdown_connections 2024-03-29T11:21:20.659 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.683+0000 7f0ead34d640 1 -- 172.21.15.86:0/1202684648 wait complete. 2024-03-29T11:21:20.661 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.684+0000 7f0ead34d640 10 monclient: build_initial_monmap 2024-03-29T11:21:20.661 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.684+0000 7f0ead34d640 10 monclient: monmap: 2024-03-29T11:21:20.661 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:21:20.661 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:20.661 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:21:20.661 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:21:20.661 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:20.661 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:20.661 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:21:20.661 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:21:20.661 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:21:20.661 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:20.662 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.685+0000 7f0ead34d640 1 librados: starting msgr at 2024-03-29T11:21:20.663 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.685+0000 7f0ead34d640 1 librados: starting objecter 2024-03-29T11:21:20.663 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.685+0000 7f0ead34d640 1 Processor -- start 2024-03-29T11:21:20.663 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.686+0000 7f0ead34d640 1 -- start start 2024-03-29T11:21:20.663 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.686+0000 7f0ead34d640 1 librados: setting wanted keys 2024-03-29T11:21:20.663 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.686+0000 7f0ead34d640 1 librados: calling monclient init 2024-03-29T11:21:20.663 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.686+0000 7f0ead34d640 10 monclient: init 2024-03-29T11:21:20.664 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.688+0000 7f0ead34d640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:20.664 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.688+0000 7f0ead34d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:20.667 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.690+0000 7f0ead34d640 10 monclient(hunting): picked mon.noname-b con 0x7f0ea80a9ac0 addr v1:172.21.15.196:6789/0 2024-03-29T11:21:20.667 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.691+0000 7f0ead34d640 10 monclient(hunting): picked mon.noname-c con 0x7f0ea80af900 addr v1:172.21.15.86:6790/0 2024-03-29T11:21:20.668 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.691+0000 7f0ead34d640 10 monclient(hunting): picked mon.noname-a con 0x7f0ea813ee90 addr v1:172.21.15.86:6789/0 2024-03-29T11:21:20.668 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.691+0000 7f0ead34d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ea81418f0 con 0x7f0ea813ee90 2024-03-29T11:21:20.668 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.691+0000 7f0ead34d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ea8142340 con 0x7f0ea80a9ac0 2024-03-29T11:21:20.668 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.691+0000 7f0ead34d640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ea8141b60 con 0x7f0ea80af900 2024-03-29T11:21:20.668 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.691+0000 7f0ead34d640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:20.668 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.691+0000 7f0ead34d640 10 monclient(hunting): authenticate will time out at 905.009099s 2024-03-29T11:21:20.671 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.694+0000 7f0ea6ffd640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f0ea80a9ac0 0x7f0ea8140560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:57847/0 (socket says 172.21.15.86:57847) 2024-03-29T11:21:20.671 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.694+0000 7f0ea6ffd640 1 -- 172.21.15.86:0/3854057675 learned_addr learned my addr 172.21.15.86:0/3854057675 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:21:20.671 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.695+0000 7f0e8f7fe640 10 client.?.objecter ms_handle_connect 0x7f0ea80af900 2024-03-29T11:21:20.671 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.695+0000 7f0e8f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:20.679 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.698+0000 7f0e8f7fe640 10 client.?.objecter ms_handle_connect 0x7f0ea813ee90 2024-03-29T11:21:20.679 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.698+0000 7f0e8f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:20.680 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.698+0000 7f0e8f7fe640 10 client.?.objecter ms_handle_connect 0x7f0ea80a9ac0 2024-03-29T11:21:20.680 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.698+0000 7f0e8f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:20.680 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.698+0000 7f0e8f7fe640 1 -- 172.21.15.86:0/3854057675 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2927247422 0 0) 0x7f0ea81418f0 con 0x7f0ea813ee90 2024-03-29T11:21:20.680 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.698+0000 7f0e8f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:20.680 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.698+0000 7f0e8f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:20.680 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.698+0000 7f0e8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:20.680 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.698+0000 7f0e8f7fe640 10 monclient(hunting): my global_id is 4260 2024-03-29T11:21:20.680 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.698+0000 7f0e8f7fe640 1 -- 172.21.15.86:0/3854057675 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0e94004eb0 con 0x7f0ea813ee90 2024-03-29T11:21:20.680 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.698+0000 7f0e8f7fe640 1 -- 172.21.15.86:0/3854057675 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 159928011 0 0) 0x7f0ea8141b60 con 0x7f0ea80af900 2024-03-29T11:21:20.680 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.698+0000 7f0e8f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:20.680 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.698+0000 7f0e8f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:20.680 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.698+0000 7f0e8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:20.680 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.698+0000 7f0e8f7fe640 10 monclient(hunting): my global_id is 4259 2024-03-29T11:21:20.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.698+0000 7f0e8f7fe640 1 -- 172.21.15.86:0/3854057675 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ea81418f0 con 0x7f0ea80af900 2024-03-29T11:21:20.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.698+0000 7f0e8f7fe640 1 -- 172.21.15.86:0/3854057675 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 717521792 0 0) 0x7f0ea8142340 con 0x7f0ea80a9ac0 2024-03-29T11:21:20.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.698+0000 7f0e8f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:20.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.698+0000 7f0e8f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:20.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.698+0000 7f0e8f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:20.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.698+0000 7f0e8f7fe640 10 monclient(hunting): my global_id is 4249 2024-03-29T11:21:20.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.698+0000 7f0e8f7fe640 1 -- 172.21.15.86:0/3854057675 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ea8141b60 con 0x7f0ea80a9ac0 2024-03-29T11:21:20.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.699+0000 7f0e8f7fe640 1 -- 172.21.15.86:0/3854057675 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2258674196 0 0) 0x7f0ea81418f0 con 0x7f0ea80af900 2024-03-29T11:21:20.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.699+0000 7f0e8f7fe640 1 -- 172.21.15.86:0/3854057675 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ea8142340 con 0x7f0ea80af900 2024-03-29T11:21:20.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.699+0000 7f0e8f7fe640 1 -- 172.21.15.86:0/3854057675 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1286200600 0 0) 0x7f0e94004eb0 con 0x7f0ea813ee90 2024-03-29T11:21:20.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.699+0000 7f0e8f7fe640 1 -- 172.21.15.86:0/3854057675 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0ea81418f0 con 0x7f0ea813ee90 2024-03-29T11:21:20.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.699+0000 7f0e8f7fe640 1 -- 172.21.15.86:0/3854057675 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3990259245 0 0) 0x7f0ea8141b60 con 0x7f0ea80a9ac0 2024-03-29T11:21:20.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.699+0000 7f0e8f7fe640 1 -- 172.21.15.86:0/3854057675 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0e94004eb0 con 0x7f0ea80a9ac0 2024-03-29T11:21:20.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.699+0000 7f0e8f7fe640 1 -- 172.21.15.86:0/3854057675 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f0e980016f0 con 0x7f0ea80af900 2024-03-29T11:21:20.682 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.699+0000 7f0e8f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:20.682 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.699+0000 7f0e8f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:20.682 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.699+0000 7f0e8f7fe640 10 monclient(hunting): dump: 2024-03-29T11:21:20.682 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:21:20.682 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:20.682 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:20.682 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:20.682 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:20.682 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:20.682 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:21:20.682 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:21:20.682 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:21:20.682 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:20.682 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.699+0000 7f0e8f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:20.683 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.699+0000 7f0e8f7fe640 1 -- 172.21.15.86:0/3854057675 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f0ea0001830 con 0x7f0ea813ee90 2024-03-29T11:21:20.683 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.699+0000 7f0e8f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:20.683 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.699+0000 7f0e8f7fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:20.683 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.699+0000 7f0e8f7fe640 10 monclient(hunting): dump: 2024-03-29T11:21:20.683 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:21:20.683 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:20.683 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:20.683 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:20.683 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:20.683 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:20.683 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:21:20.683 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:21:20.683 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:21:20.683 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:20.683 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.699+0000 7f0e8f7fe640 1 -- 172.21.15.86:0/3854057675 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f0e9c001750 con 0x7f0ea80a9ac0 2024-03-29T11:21:20.684 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.699+0000 7f0e8f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:20.684 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.699+0000 7f0e8f7fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:20.684 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.699+0000 7f0e8f7fe640 10 monclient(hunting): dump: 2024-03-29T11:21:20.684 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:21:20.684 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:20.684 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:20.684 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:20.684 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:20.684 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:20.684 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:21:20.684 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:21:20.684 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:21:20.684 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:20.684 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.700+0000 7f0e8f7fe640 1 -- 172.21.15.86:0/3854057675 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2635780917 0 0) 0x7f0e94004eb0 con 0x7f0ea80a9ac0 2024-03-29T11:21:20.684 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.700+0000 7f0e8f7fe640 1 -- 172.21.15.86:0/3854057675 >> v1:172.21.15.86:6790/0 conn(0x7f0ea80af900 legacy=0x7f0ea813c0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:20.685 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.700+0000 7f0e8f7fe640 1 -- 172.21.15.86:0/3854057675 >> v1:172.21.15.86:6789/0 conn(0x7f0ea813ee90 legacy=0x7f0ea813c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:20.685 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.700+0000 7f0e8f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:21:20.685 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.700+0000 7f0e8f7fe640 1 monclient: found mon.b 2024-03-29T11:21:20.685 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.700+0000 7f0e8f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:20.685 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.700+0000 7f0e8f7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:21:20.685 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.700+0000 7f0e8f7fe640 1 -- 172.21.15.86:0/3854057675 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ea812f5a0 con 0x7f0ea80a9ac0 2024-03-29T11:21:20.685 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.700+0000 7f0e8f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:21:20.685 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.700+0000 7f0e8f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:21:20.685 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.700+0000 7f0e8f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:20.685 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.701+0000 7f0e8f7fe640 1 -- 172.21.15.86:0/3854057675 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0e9c0018b0 con 0x7f0ea80a9ac0 2024-03-29T11:21:20.685 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.701+0000 7f0ead34d640 5 monclient: authenticate success, global_id 4249 2024-03-29T11:21:20.685 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.701+0000 7f0ead34d640 10 monclient: _renew_subs 2024-03-29T11:21:20.685 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.701+0000 7f0ead34d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:21:20.685 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.701+0000 7f0ead34d640 1 -- 172.21.15.86:0/3854057675 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ea812f850 con 0x7f0ea80a9ac0 2024-03-29T11:21:20.685 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.701+0000 7f0ead34d640 10 client.4249.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:20.686 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.701+0000 7f0ead34d640 10 monclient: _renew_subs 2024-03-29T11:21:20.686 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.701+0000 7f0ead34d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:21:20.686 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.701+0000 7f0ead34d640 1 -- 172.21.15.86:0/3854057675 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ea81e2dc0 con 0x7f0ea80a9ac0 2024-03-29T11:21:20.686 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.701+0000 7f0ead34d640 1 librados: init done 2024-03-29T11:21:20.686 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.701+0000 7f0e8f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:20.686 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.701+0000 7f0e8f7fe640 1 -- 172.21.15.86:0/3854057675 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f0e9c003790 con 0x7f0ea80a9ac0 2024-03-29T11:21:20.686 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.701+0000 7f0e8f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:20.686 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.701+0000 7f0e8f7fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:20.686 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.701+0000 7f0e8f7fe640 10 monclient: dump: 2024-03-29T11:21:20.686 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:21:20.686 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:20.686 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:20.686 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:20.686 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:20.686 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:20.687 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:21:20.687 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:21:20.687 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:21:20.687 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:20.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.703+0000 7f0ea4ff9640 4 set_mon_vals no callback set 2024-03-29T11:21:20.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.703+0000 7f0ead34d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:20.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.704+0000 7f0e8f7fe640 1 -- 172.21.15.86:0/3854057675 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f0e9c0038f0 con 0x7f0ea80a9ac0 2024-03-29T11:21:20.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.705+0000 7f0ead34d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:20.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.705+0000 7f0ead34d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:21:20.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.705+0000 7f0ead34d640 1 -- 172.21.15.86:0/3854057675 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e88005080 con 0x7f0ea80a9ac0 2024-03-29T11:21:20.689 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.705+0000 7f0e8f7fe640 1 -- 172.21.15.86:0/3854057675 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f0e9c084890 con 0x7f0ea80a9ac0 2024-03-29T11:21:20.689 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.705+0000 7f0ea67fc640 10 monclient: get_auth_request con 0x7f0e9406e300 auth_method 0 2024-03-29T11:21:20.689 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.705+0000 7f0e8f7fe640 10 client.4249.objecter ms_dispatch 0x7f0ea8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:20.689 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.706+0000 7f0e8f7fe640 3 client.4249.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:20.689 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.706+0000 7f0e8f7fe640 3 client.4249.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:20.689 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.706+0000 7f0e8f7fe640 20 client.4249.objecter dump_active .. 0 homeless 2024-03-29T11:21:20.689 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.706+0000 7f0e8f7fe640 10 client.4249.objecter ms_handle_connect 0x7f0e9406e300 2024-03-29T11:21:20.692 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.715+0000 7f0e8f7fe640 1 -- 172.21.15.86:0/3854057675 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0e9c050170 con 0x7f0ea80a9ac0 2024-03-29T11:21:20.692 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.715+0000 7f0e8f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:20.692 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:20.715+0000 7f0e8f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:21.023 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:21.046+0000 7f0ead34d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:21.023 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:21.046+0000 7f0ead34d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:21.023 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:21.046+0000 7f0ead34d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:21:21.023 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:21.046+0000 7f0ead34d640 1 -- 172.21.15.86:0/3854057675 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0e88005330 con 0x7f0ea80a9ac0 2024-03-29T11:21:21.024 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:21.047+0000 7f0e8f7fe640 1 -- 172.21.15.86:0/3854057675 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f0e9c0520a0 con 0x7f0ea80a9ac0 2024-03-29T11:21:21.024 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:21.047+0000 7f0e8f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:21.024 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:21.047+0000 7f0e8f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:21.025 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:21:21.025 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:21.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:21.052+0000 7f0ead34d640 10 librados: watch_flush enter 2024-03-29T11:21:21.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:21.052+0000 7f0ead34d640 10 librados: watch_flush exit 2024-03-29T11:21:21.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:21.052+0000 7f0ead34d640 20 client.4249.objecter shutdown clearing up homeless session... 2024-03-29T11:21:21.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:21.052+0000 7f0ead34d640 10 client.4249.objecter successfully canceled tick 2024-03-29T11:21:21.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:21.053+0000 7f0ead34d640 1 -- 172.21.15.86:0/3854057675 >> v1:172.21.15.196:6816/588110318 conn(0x7f0e9406e300 legacy=0x7f0e940707c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:21.030 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:21.053+0000 7f0ead34d640 10 monclient: shutdown 2024-03-29T11:21:21.030 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:21.053+0000 7f0ead34d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:21.030 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:21.053+0000 7f0ead34d640 1 -- 172.21.15.86:0/3854057675 >> v1:172.21.15.196:6789/0 conn(0x7f0ea80a9ac0 legacy=0x7f0ea8140560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:21.030 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:21.053+0000 7f0ead34d640 1 -- 172.21.15.86:0/3854057675 shutdown_connections 2024-03-29T11:21:21.030 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:21.053+0000 7f0ead34d640 1 -- 172.21.15.86:0/3854057675 >> 172.21.15.86:0/3854057675 conn(0x7f0ea80b35e0 msgr2=0x7f0ea80aceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:21.030 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:21.053+0000 7f0ead34d640 1 -- 172.21.15.86:0/3854057675 shutdown_connections 2024-03-29T11:21:21.030 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:21.053+0000 7f0ead34d640 1 -- 172.21.15.86:0/3854057675 wait complete. 2024-03-29T11:21:21.030 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:21.053+0000 7f0ead34d640 1 librados: shutdown 2024-03-29T11:21:21.047 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:27.854 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:28.013 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.035+0000 7f30348d5640 10 monclient: get_monmap_and_config 2024-03-29T11:21:28.013 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.035+0000 7f30348d5640 10 monclient: build_initial_monmap 2024-03-29T11:21:28.013 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.035+0000 7f30348d5640 10 monclient: monmap: 2024-03-29T11:21:28.013 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:21:28.013 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:28.013 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:21:28.036607+0000 2024-03-29T11:21:28.013 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:21:28.036607+0000 2024-03-29T11:21:28.013 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:28.013 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:28.013 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:21:28.014 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:21:28.014 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:21:28.014 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:28.014 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.036+0000 7f30348d5640 1 Processor -- start 2024-03-29T11:21:28.014 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.036+0000 7f30348d5640 1 -- start start 2024-03-29T11:21:28.014 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.036+0000 7f30348d5640 10 monclient: init 2024-03-29T11:21:28.014 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.037+0000 7f30348d5640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:28.014 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.037+0000 7f30348d5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:28.014 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.037+0000 7f30348d5640 10 monclient(hunting): picked mon.noname-b con 0x7f30300ae220 addr v1:172.21.15.86:6790/0 2024-03-29T11:21:28.015 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.037+0000 7f30348d5640 10 monclient(hunting): picked mon.noname-a con 0x7f3030072f10 addr v1:172.21.15.86:6789/0 2024-03-29T11:21:28.015 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.037+0000 7f30348d5640 10 monclient(hunting): picked mon.noname-c con 0x7f30300acea0 addr v1:172.21.15.196:6789/0 2024-03-29T11:21:28.015 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.037+0000 7f30348d5640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30300aa9f0 con 0x7f3030072f10 2024-03-29T11:21:28.015 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.037+0000 7f30348d5640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30300ac4c0 con 0x7f30300acea0 2024-03-29T11:21:28.015 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.037+0000 7f30348d5640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30300ac7a0 con 0x7f30300ae220 2024-03-29T11:21:28.015 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.037+0000 7f30348d5640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:28.015 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.037+0000 7f30348d5640 10 monclient(hunting): authenticate will time out at 912.355468s 2024-03-29T11:21:28.018 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.042+0000 7f302e59c640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f3030072f10 0x7f3030141d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:37343/0 (socket says 172.21.15.86:37343) 2024-03-29T11:21:28.018 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.042+0000 7f302e59c640 1 -- 172.21.15.86:0/2115468751 learned_addr learned my addr 172.21.15.86:0/2115468751 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:21:28.021 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.045+0000 7f302dd9b640 1 -- 172.21.15.86:0/2115468751 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3129331746 0 0) 0x7f30300aa9f0 con 0x7f3030072f10 2024-03-29T11:21:28.022 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.045+0000 7f302dd9b640 10 monclient(hunting): _negotiate 2024-03-29T11:21:28.022 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.045+0000 7f302dd9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:28.022 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.045+0000 7f302dd9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:28.022 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.045+0000 7f302dd9b640 10 monclient(hunting): my global_id is 4263 2024-03-29T11:21:28.024 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.045+0000 7f302dd9b640 1 -- 172.21.15.86:0/2115468751 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3018003520 con 0x7f3030072f10 2024-03-29T11:21:28.024 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.047+0000 7f302dd9b640 1 -- 172.21.15.86:0/2115468751 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 141259462 0 0) 0x7f3018003520 con 0x7f3030072f10 2024-03-29T11:21:28.024 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.047+0000 7f302dd9b640 1 -- 172.21.15.86:0/2115468751 >> v1:172.21.15.86:6790/0 conn(0x7f30300ae220 legacy=0x7f3030141450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:28.024 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.047+0000 7f302dd9b640 1 -- 172.21.15.86:0/2115468751 >> v1:172.21.15.196:6789/0 conn(0x7f30300acea0 legacy=0x7f30300ad2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:28.024 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.047+0000 7f302dd9b640 10 monclient: _finish_hunting 0 2024-03-29T11:21:28.024 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.047+0000 7f302dd9b640 1 monclient: found mon.noname-a 2024-03-29T11:21:28.024 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.047+0000 7f302dd9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:28.024 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.047+0000 7f302dd9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.86:6789/0 2024-03-29T11:21:28.024 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.047+0000 7f302dd9b640 1 -- 172.21.15.86:0/2115468751 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30300ab5a0 con 0x7f3030072f10 2024-03-29T11:21:28.024 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.047+0000 7f302dd9b640 10 monclient: _finish_auth 0 2024-03-29T11:21:28.024 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.047+0000 7f302dd9b640 10 monclient: _check_auth_tickets 2024-03-29T11:21:28.024 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.047+0000 7f302dd9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:28.025 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.047+0000 7f30348d5640 5 monclient: authenticate success, global_id 4263 2024-03-29T11:21:28.025 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.047+0000 7f30348d5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:28.025 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.047+0000 7f302dd9b640 1 -- 172.21.15.86:0/2115468751 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f3028001a10 con 0x7f3030072f10 2024-03-29T11:21:28.025 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.047+0000 7f302dd9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:28.025 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.047+0000 7f302dd9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:28.025 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.047+0000 7f302dd9b640 10 monclient: dump: 2024-03-29T11:21:28.025 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:21:28.025 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:28.025 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:28.025 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:28.025 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:28.025 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:28.025 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:21:28.025 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:21:28.025 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:21:28.025 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:28.026 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.047+0000 7f30348d5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:28.026 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.047+0000 7f302dd9b640 1 -- 172.21.15.86:0/2115468751 <== mon.0 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3028001ca0 con 0x7f3030072f10 2024-03-29T11:21:28.026 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.047+0000 7f302dd9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:28.026 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.047+0000 7f30348d5640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:28.026 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.047+0000 7f30348d5640 4 set_mon_vals no callback set 2024-03-29T11:21:28.026 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.048+0000 7f30348d5640 10 monclient: shutdown 2024-03-29T11:21:28.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.048+0000 7f30348d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:28.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.048+0000 7f30348d5640 1 -- 172.21.15.86:0/2115468751 >> v1:172.21.15.86:6789/0 conn(0x7f3030072f10 legacy=0x7f3030141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:28.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.049+0000 7f30348d5640 1 -- 172.21.15.86:0/2115468751 shutdown_connections 2024-03-29T11:21:28.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.049+0000 7f30348d5640 1 -- 172.21.15.86:0/2115468751 >> 172.21.15.86:0/2115468751 conn(0x7f30300b3a30 msgr2=0x7f30300b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:28.030 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.050+0000 7f30348d5640 1 -- 172.21.15.86:0/2115468751 shutdown_connections 2024-03-29T11:21:28.030 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.050+0000 7f30348d5640 1 -- 172.21.15.86:0/2115468751 wait complete. 2024-03-29T11:21:28.030 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.051+0000 7f30348d5640 10 monclient: build_initial_monmap 2024-03-29T11:21:28.030 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.051+0000 7f30348d5640 10 monclient: monmap: 2024-03-29T11:21:28.030 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:21:28.030 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:28.030 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:21:28.030 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:21:28.030 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:28.030 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:28.031 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:21:28.031 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:21:28.031 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:21:28.031 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:28.031 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.052+0000 7f30348d5640 1 librados: starting msgr at 2024-03-29T11:21:28.031 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.052+0000 7f30348d5640 1 librados: starting objecter 2024-03-29T11:21:28.031 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.052+0000 7f30348d5640 1 Processor -- start 2024-03-29T11:21:28.031 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.052+0000 7f30348d5640 1 -- start start 2024-03-29T11:21:28.031 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.052+0000 7f30348d5640 1 librados: setting wanted keys 2024-03-29T11:21:28.031 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.052+0000 7f30348d5640 1 librados: calling monclient init 2024-03-29T11:21:28.031 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.052+0000 7f30348d5640 10 monclient: init 2024-03-29T11:21:28.031 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.053+0000 7f30348d5640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:28.031 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.053+0000 7f30348d5640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:21:28.031 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.053+0000 7f30348d5640 10 monclient(hunting): picked mon.noname-a con 0x7f3030072f10 addr v1:172.21.15.86:6789/0 2024-03-29T11:21:28.033 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.056+0000 7f30348d5640 10 monclient(hunting): picked mon.noname-c con 0x7f30300acea0 addr v1:172.21.15.86:6790/0 2024-03-29T11:21:28.033 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.056+0000 7f302e59c640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f30300acea0 0x7f3030136430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:47147/0 (socket says 172.21.15.86:47147) 2024-03-29T11:21:28.033 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.056+0000 7f302e59c640 1 -- 172.21.15.86:0/3515057722 learned_addr learned my addr 172.21.15.86:0/3515057722 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:21:28.036 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.059+0000 7f30177fe640 10 client.?.objecter ms_handle_connect 0x7f3030072f10 2024-03-29T11:21:28.036 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.059+0000 7f30177fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:28.038 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.060+0000 7f30348d5640 10 monclient(hunting): picked mon.noname-b con 0x7f30300ae220 addr v1:172.21.15.196:6789/0 2024-03-29T11:21:28.038 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.060+0000 7f30348d5640 1 -- 172.21.15.86:0/3515057722 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3030071ef0 con 0x7f3030072f10 2024-03-29T11:21:28.038 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.060+0000 7f30348d5640 1 -- 172.21.15.86:0/3515057722 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3030136010 con 0x7f30300ae220 2024-03-29T11:21:28.038 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.060+0000 7f30348d5640 1 -- 172.21.15.86:0/3515057722 --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3030153b90 con 0x7f30300acea0 2024-03-29T11:21:28.038 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.060+0000 7f30348d5640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:28.039 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.060+0000 7f30348d5640 10 monclient(hunting): authenticate will time out at 912.378223s 2024-03-29T11:21:28.039 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.060+0000 7f30177fe640 10 client.?.objecter ms_handle_connect 0x7f30300acea0 2024-03-29T11:21:28.039 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.060+0000 7f30177fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:28.039 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.061+0000 7f30177fe640 1 -- 172.21.15.86:0/3515057722 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2431321344 0 0) 0x7f3030153b90 con 0x7f30300acea0 2024-03-29T11:21:28.039 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.061+0000 7f30177fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:28.039 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.062+0000 7f30177fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:28.039 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.062+0000 7f30177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:28.039 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.062+0000 7f30177fe640 10 monclient(hunting): my global_id is 4265 2024-03-29T11:21:28.039 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.062+0000 7f30177fe640 1 -- 172.21.15.86:0/3515057722 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f301c004eb0 con 0x7f30300acea0 2024-03-29T11:21:28.039 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.062+0000 7f30177fe640 10 client.?.objecter ms_handle_connect 0x7f30300ae220 2024-03-29T11:21:28.039 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.062+0000 7f30177fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:28.040 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.062+0000 7f30177fe640 1 -- 172.21.15.86:0/3515057722 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 10040460 0 0) 0x7f3030136010 con 0x7f30300ae220 2024-03-29T11:21:28.040 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.062+0000 7f30177fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:28.040 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.062+0000 7f30177fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:28.040 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.062+0000 7f30177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:28.040 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.062+0000 7f30177fe640 10 monclient(hunting): my global_id is 4252 2024-03-29T11:21:28.040 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.062+0000 7f30177fe640 1 -- 172.21.15.86:0/3515057722 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3030153b90 con 0x7f30300ae220 2024-03-29T11:21:28.040 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.062+0000 7f30177fe640 1 -- 172.21.15.86:0/3515057722 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2119465084 0 0) 0x7f301c004eb0 con 0x7f30300acea0 2024-03-29T11:21:28.040 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.062+0000 7f30177fe640 1 -- 172.21.15.86:0/3515057722 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3030136010 con 0x7f30300acea0 2024-03-29T11:21:28.040 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.062+0000 7f30177fe640 1 -- 172.21.15.86:0/3515057722 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f3028004e60 con 0x7f30300acea0 2024-03-29T11:21:28.040 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.062+0000 7f30177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:28.041 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.062+0000 7f30177fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:28.041 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.062+0000 7f30177fe640 10 monclient(hunting): dump: 2024-03-29T11:21:28.041 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:21:28.041 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:28.041 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:28.041 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:28.041 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:28.041 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:28.041 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:21:28.041 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:21:28.041 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:21:28.041 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:28.041 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.062+0000 7f30177fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:28.041 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.062+0000 7f30177fe640 1 -- 172.21.15.86:0/3515057722 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2831424903 0 0) 0x7f3030071ef0 con 0x7f3030072f10 2024-03-29T11:21:28.041 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.062+0000 7f30177fe640 10 monclient(hunting): _negotiate 2024-03-29T11:21:28.041 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.062+0000 7f30177fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:28.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.062+0000 7f30177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:28.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.062+0000 7f30177fe640 10 monclient(hunting): my global_id is 4266 2024-03-29T11:21:28.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.062+0000 7f30177fe640 1 -- 172.21.15.86:0/3515057722 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f301c004eb0 con 0x7f3030072f10 2024-03-29T11:21:28.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.062+0000 7f30177fe640 1 -- 172.21.15.86:0/3515057722 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3115828592 0 0) 0x7f3030136010 con 0x7f30300acea0 2024-03-29T11:21:28.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.063+0000 7f30177fe640 1 -- 172.21.15.86:0/3515057722 >> v1:172.21.15.196:6789/0 conn(0x7f30300ae220 legacy=0x7f30301eed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:28.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.063+0000 7f30177fe640 1 -- 172.21.15.86:0/3515057722 >> v1:172.21.15.86:6789/0 conn(0x7f3030072f10 legacy=0x7f30300700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:28.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.063+0000 7f30177fe640 10 monclient: _finish_hunting 0 2024-03-29T11:21:28.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.063+0000 7f30177fe640 1 monclient: found mon.c 2024-03-29T11:21:28.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.063+0000 7f30177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:28.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.063+0000 7f30177fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:21:28.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.063+0000 7f30177fe640 1 -- 172.21.15.86:0/3515057722 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3030153d20 con 0x7f30300acea0 2024-03-29T11:21:28.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.063+0000 7f30177fe640 10 monclient: _finish_auth 0 2024-03-29T11:21:28.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.063+0000 7f30177fe640 10 monclient: _check_auth_tickets 2024-03-29T11:21:28.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.063+0000 7f30177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:28.043 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.063+0000 7f30177fe640 1 -- 172.21.15.86:0/3515057722 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3028005270 con 0x7f30300acea0 2024-03-29T11:21:28.043 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.063+0000 7f30177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:28.043 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.063+0000 7f30177fe640 1 -- 172.21.15.86:0/3515057722 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f30280056c0 con 0x7f30300acea0 2024-03-29T11:21:28.043 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.063+0000 7f30177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:28.043 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.063+0000 7f30177fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:28.043 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.063+0000 7f30177fe640 10 monclient: dump: 2024-03-29T11:21:28.043 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:21:28.043 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:28.043 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:28.043 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:28.043 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:28.043 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:28.043 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:21:28.043 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:21:28.043 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:21:28.044 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:28.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.063+0000 7f30348d5640 5 monclient: authenticate success, global_id 4265 2024-03-29T11:21:28.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.063+0000 7f30348d5640 10 monclient: _renew_subs 2024-03-29T11:21:28.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.063+0000 7f30348d5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:21:28.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.063+0000 7f30348d5640 1 -- 172.21.15.86:0/3515057722 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3030134c20 con 0x7f30300acea0 2024-03-29T11:21:28.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.063+0000 7f30348d5640 10 client.4265.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:28.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.063+0000 7f30348d5640 10 monclient: _renew_subs 2024-03-29T11:21:28.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.063+0000 7f30348d5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:21:28.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.063+0000 7f30348d5640 1 -- 172.21.15.86:0/3515057722 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3030072660 con 0x7f30300acea0 2024-03-29T11:21:28.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.063+0000 7f30348d5640 1 librados: init done 2024-03-29T11:21:28.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.064+0000 7f302d59a640 4 set_mon_vals no callback set 2024-03-29T11:21:28.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.066+0000 7f30348d5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:28.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.066+0000 7f30348d5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:28.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.066+0000 7f30348d5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:21:28.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.066+0000 7f30348d5640 1 -- 172.21.15.86:0/3515057722 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3010005080 con 0x7f30300acea0 2024-03-29T11:21:28.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.068+0000 7f30177fe640 1 -- 172.21.15.86:0/3515057722 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f3028001e90 con 0x7f30300acea0 2024-03-29T11:21:28.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.068+0000 7f30177fe640 1 -- 172.21.15.86:0/3515057722 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f3028085040 con 0x7f30300acea0 2024-03-29T11:21:28.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.068+0000 7f30177fe640 10 client.4265.objecter ms_dispatch 0x7f3030000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:28.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.068+0000 7f30177fe640 3 client.4265.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:28.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.068+0000 7f30177fe640 3 client.4265.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:28.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.069+0000 7f30177fe640 20 client.4265.objecter dump_active .. 0 homeless 2024-03-29T11:21:28.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.069+0000 7f302ed9d640 10 monclient: get_auth_request con 0x7f301c06de80 auth_method 0 2024-03-29T11:21:28.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.069+0000 7f30177fe640 10 client.4265.objecter ms_handle_connect 0x7f301c06de80 2024-03-29T11:21:28.054 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.077+0000 7f30177fe640 1 -- 172.21.15.86:0/3515057722 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3028050920 con 0x7f30300acea0 2024-03-29T11:21:28.054 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.077+0000 7f30177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:28.054 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.077+0000 7f30177fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:28.378 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.399+0000 7f30348d5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:28.378 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.399+0000 7f30348d5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:28.378 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.399+0000 7f30348d5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:21:28.378 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.399+0000 7f30348d5640 1 -- 172.21.15.86:0/3515057722 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3010005330 con 0x7f30300acea0 2024-03-29T11:21:28.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.402+0000 7f30177fe640 1 -- 172.21.15.86:0/3515057722 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f3028052850 con 0x7f30300acea0 2024-03-29T11:21:28.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.402+0000 7f30177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:28.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.402+0000 7f30177fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:28.379 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:21:28.379 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:28.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.406+0000 7f30348d5640 10 librados: watch_flush enter 2024-03-29T11:21:28.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.406+0000 7f30348d5640 10 librados: watch_flush exit 2024-03-29T11:21:28.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.406+0000 7f30348d5640 20 client.4265.objecter shutdown clearing up homeless session... 2024-03-29T11:21:28.385 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.406+0000 7f30348d5640 10 client.4265.objecter successfully canceled tick 2024-03-29T11:21:28.385 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.406+0000 7f30348d5640 1 -- 172.21.15.86:0/3515057722 >> v1:172.21.15.196:6816/588110318 conn(0x7f301c06de80 legacy=0x7f301c070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:28.385 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.406+0000 7f30348d5640 10 monclient: shutdown 2024-03-29T11:21:28.385 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.406+0000 7f30348d5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:28.385 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.406+0000 7f30348d5640 1 -- 172.21.15.86:0/3515057722 >> v1:172.21.15.86:6790/0 conn(0x7f30300acea0 legacy=0x7f3030136430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:28.385 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.406+0000 7f30348d5640 1 -- 172.21.15.86:0/3515057722 shutdown_connections 2024-03-29T11:21:28.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.406+0000 7f30348d5640 1 -- 172.21.15.86:0/3515057722 >> 172.21.15.86:0/3515057722 conn(0x7f30300b3a30 msgr2=0x7f30300af1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:28.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.407+0000 7f30348d5640 1 -- 172.21.15.86:0/3515057722 shutdown_connections 2024-03-29T11:21:28.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.408+0000 7f30348d5640 1 -- 172.21.15.86:0/3515057722 wait complete. 2024-03-29T11:21:28.388 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:28.408+0000 7f30348d5640 1 librados: shutdown 2024-03-29T11:21:28.402 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:35.209 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:35.364 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.386+0000 7f7c0f9e2640 10 monclient: get_monmap_and_config 2024-03-29T11:21:35.364 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.386+0000 7f7c0f9e2640 10 monclient: build_initial_monmap 2024-03-29T11:21:35.364 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.386+0000 7f7c0f9e2640 10 monclient: monmap: 2024-03-29T11:21:35.364 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:21:35.364 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:35.364 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:21:35.387363+0000 2024-03-29T11:21:35.364 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:21:35.387363+0000 2024-03-29T11:21:35.364 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:35.364 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:35.364 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:21:35.365 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:21:35.365 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:21:35.365 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:35.365 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.387+0000 7f7c0f9e2640 1 Processor -- start 2024-03-29T11:21:35.365 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.388+0000 7f7c0f9e2640 1 -- start start 2024-03-29T11:21:35.365 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.388+0000 7f7c0f9e2640 10 monclient: init 2024-03-29T11:21:35.365 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.388+0000 7f7c0f9e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:35.365 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.388+0000 7f7c0f9e2640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:21:35.366 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.389+0000 7f7c0f9e2640 10 monclient(hunting): picked mon.noname-c con 0x7f7c080a9ac0 addr v1:172.21.15.196:6789/0 2024-03-29T11:21:35.366 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.389+0000 7f7c0f9e2640 10 monclient(hunting): picked mon.noname-a con 0x7f7c0813ee90 addr v1:172.21.15.86:6789/0 2024-03-29T11:21:35.369 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.389+0000 7f7c0f9e2640 10 monclient(hunting): picked mon.noname-b con 0x7f7c080af900 addr v1:172.21.15.86:6790/0 2024-03-29T11:21:35.369 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.390+0000 7f7c0f9e2640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c080afd10 con 0x7f7c0813ee90 2024-03-29T11:21:35.369 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.390+0000 7f7c0f9e2640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c080af6f0 con 0x7f7c080a9ac0 2024-03-29T11:21:35.369 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.390+0000 7f7c0f9e2640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c080aac80 con 0x7f7c080af900 2024-03-29T11:21:35.369 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.390+0000 7f7c0f9e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:35.369 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.390+0000 7f7c0f9e2640 10 monclient(hunting): authenticate will time out at 919.708738s 2024-03-29T11:21:35.370 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.394+0000 7f7c0cf7d640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f7c0813ee90 0x7f7c0813b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:41009/0 (socket says 172.21.15.86:41009) 2024-03-29T11:21:35.370 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.394+0000 7f7c0cf7d640 1 -- 172.21.15.86:0/4052167794 learned_addr learned my addr 172.21.15.86:0/4052167794 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:21:35.372 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.395+0000 7f7bfffff640 1 -- 172.21.15.86:0/4052167794 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1197593437 0 0) 0x7f7c080afd10 con 0x7f7c0813ee90 2024-03-29T11:21:35.372 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.395+0000 7f7bfffff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:35.372 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.396+0000 7f7bfffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:35.372 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.396+0000 7f7bfffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:35.372 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.396+0000 7f7bfffff640 10 monclient(hunting): my global_id is 4269 2024-03-29T11:21:35.375 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.396+0000 7f7bfffff640 1 -- 172.21.15.86:0/4052167794 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bf4003520 con 0x7f7c0813ee90 2024-03-29T11:21:35.375 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.396+0000 7f7bfffff640 1 -- 172.21.15.86:0/4052167794 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2000524211 0 0) 0x7f7c080aac80 con 0x7f7c080af900 2024-03-29T11:21:35.375 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.396+0000 7f7bfffff640 10 monclient(hunting): _negotiate 2024-03-29T11:21:35.375 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.396+0000 7f7bfffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:35.375 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.396+0000 7f7bfffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:35.375 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.396+0000 7f7bfffff640 10 monclient(hunting): my global_id is 4268 2024-03-29T11:21:35.375 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.396+0000 7f7bfffff640 1 -- 172.21.15.86:0/4052167794 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c080afd10 con 0x7f7c080af900 2024-03-29T11:21:35.376 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.397+0000 7f7bfffff640 1 -- 172.21.15.86:0/4052167794 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1015335518 0 0) 0x7f7bf4003520 con 0x7f7c0813ee90 2024-03-29T11:21:35.376 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.397+0000 7f7bfffff640 1 -- 172.21.15.86:0/4052167794 >> v1:172.21.15.86:6790/0 conn(0x7f7c080af900 legacy=0x7f7c08149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:35.376 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.397+0000 7f7bfffff640 1 -- 172.21.15.86:0/4052167794 >> v1:172.21.15.196:6789/0 conn(0x7f7c080a9ac0 legacy=0x7f7c080adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:35.376 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.397+0000 7f7bfffff640 10 monclient: _finish_hunting 0 2024-03-29T11:21:35.376 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.397+0000 7f7bfffff640 1 monclient: found mon.noname-a 2024-03-29T11:21:35.376 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.397+0000 7f7bfffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:35.376 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.397+0000 7f7bfffff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.86:6789/0 2024-03-29T11:21:35.376 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.397+0000 7f7bfffff640 1 -- 172.21.15.86:0/4052167794 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c080aaf60 con 0x7f7c0813ee90 2024-03-29T11:21:35.376 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.397+0000 7f7bfffff640 10 monclient: _finish_auth 0 2024-03-29T11:21:35.376 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.397+0000 7f7bfffff640 10 monclient: _check_auth_tickets 2024-03-29T11:21:35.376 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.397+0000 7f7bfffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:35.376 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.397+0000 7f7bfffff640 1 -- 172.21.15.86:0/4052167794 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f7c00001a10 con 0x7f7c0813ee90 2024-03-29T11:21:35.376 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.397+0000 7f7bfffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:35.377 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.397+0000 7f7bfffff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:35.377 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.397+0000 7f7bfffff640 10 monclient: dump: 2024-03-29T11:21:35.377 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:21:35.377 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:35.377 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:35.377 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:35.377 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:35.377 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:35.377 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:21:35.377 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:21:35.377 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:21:35.377 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:35.377 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.398+0000 7f7c0f9e2640 5 monclient: authenticate success, global_id 4269 2024-03-29T11:21:35.377 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.398+0000 7f7c0f9e2640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:35.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.399+0000 7f7bfffff640 1 -- 172.21.15.86:0/4052167794 <== mon.0 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c00001b70 con 0x7f7c0813ee90 2024-03-29T11:21:35.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.399+0000 7f7bfffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:35.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.399+0000 7f7bfffff640 1 -- 172.21.15.86:0/4052167794 <== mon.0 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f7c00001fc0 con 0x7f7c0813ee90 2024-03-29T11:21:35.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.399+0000 7f7bfffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:35.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.399+0000 7f7bfffff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:35.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.399+0000 7f7bfffff640 10 monclient: dump: 2024-03-29T11:21:35.379 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:21:35.380 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:35.380 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:35.380 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:35.380 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:35.380 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:35.380 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:21:35.380 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:21:35.380 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:21:35.380 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:35.380 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.399+0000 7f7c0f9e2640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:35.380 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.399+0000 7f7c0f9e2640 4 set_mon_vals no callback set 2024-03-29T11:21:35.380 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.400+0000 7f7c0f9e2640 10 monclient: shutdown 2024-03-29T11:21:35.380 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.400+0000 7f7c0f9e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:35.380 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.400+0000 7f7c0f9e2640 1 -- 172.21.15.86:0/4052167794 >> v1:172.21.15.86:6789/0 conn(0x7f7c0813ee90 legacy=0x7f7c0813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:35.380 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.401+0000 7f7c0f9e2640 1 -- 172.21.15.86:0/4052167794 shutdown_connections 2024-03-29T11:21:35.381 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.401+0000 7f7c0f9e2640 1 -- 172.21.15.86:0/4052167794 >> 172.21.15.86:0/4052167794 conn(0x7f7c080b35e0 msgr2=0x7f7c080b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:35.381 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.402+0000 7f7c0f9e2640 1 -- 172.21.15.86:0/4052167794 shutdown_connections 2024-03-29T11:21:35.381 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.402+0000 7f7c0f9e2640 1 -- 172.21.15.86:0/4052167794 wait complete. 2024-03-29T11:21:35.382 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.403+0000 7f7c0f9e2640 10 monclient: build_initial_monmap 2024-03-29T11:21:35.382 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.404+0000 7f7c0f9e2640 10 monclient: monmap: 2024-03-29T11:21:35.382 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:21:35.382 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:35.382 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:21:35.382 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:21:35.382 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:35.382 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:35.382 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:21:35.382 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:21:35.383 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:21:35.383 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:35.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.404+0000 7f7c0f9e2640 1 librados: starting msgr at 2024-03-29T11:21:35.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.404+0000 7f7c0f9e2640 1 librados: starting objecter 2024-03-29T11:21:35.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.404+0000 7f7c0f9e2640 1 Processor -- start 2024-03-29T11:21:35.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.404+0000 7f7c0f9e2640 1 -- start start 2024-03-29T11:21:35.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.404+0000 7f7c0f9e2640 1 librados: setting wanted keys 2024-03-29T11:21:35.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.404+0000 7f7c0f9e2640 1 librados: calling monclient init 2024-03-29T11:21:35.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.404+0000 7f7c0f9e2640 10 monclient: init 2024-03-29T11:21:35.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.405+0000 7f7c0f9e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:35.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.405+0000 7f7c0f9e2640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:21:35.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.409+0000 7f7c0f9e2640 10 monclient(hunting): picked mon.noname-a con 0x7f7c080a9ac0 addr v1:172.21.15.86:6789/0 2024-03-29T11:21:35.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.409+0000 7f7c0f9e2640 10 monclient(hunting): picked mon.noname-b con 0x7f7c080af900 addr v1:172.21.15.196:6789/0 2024-03-29T11:21:35.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.409+0000 7f7c0f9e2640 10 monclient(hunting): picked mon.noname-c con 0x7f7c08071d70 addr v1:172.21.15.86:6790/0 2024-03-29T11:21:35.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.409+0000 7f7c0f9e2640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c08142660 con 0x7f7c080a9ac0 2024-03-29T11:21:35.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.409+0000 7f7c0f9e2640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c081d6b60 con 0x7f7c080af900 2024-03-29T11:21:35.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.409+0000 7f7c0f9e2640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c081d6cf0 con 0x7f7c08071d70 2024-03-29T11:21:35.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.409+0000 7f7c0f9e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:35.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.409+0000 7f7c0f9e2640 10 monclient(hunting): authenticate will time out at 919.727590s 2024-03-29T11:21:35.387 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.411+0000 7f7c0d77e640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f7c080a9ac0 0x7f7c081d6450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:41139/0 (socket says 172.21.15.86:41139) 2024-03-29T11:21:35.387 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.411+0000 7f7c0d77e640 1 -- 172.21.15.86:0/2662079136 learned_addr learned my addr 172.21.15.86:0/2662079136 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:21:35.389 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.412+0000 7f7bfdffb640 10 client.?.objecter ms_handle_connect 0x7f7c080a9ac0 2024-03-29T11:21:35.389 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.412+0000 7f7bfdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:35.389 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.412+0000 7f7bfdffb640 1 -- 172.21.15.86:0/2662079136 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2753883932 0 0) 0x7f7c08142660 con 0x7f7c080a9ac0 2024-03-29T11:21:35.389 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.412+0000 7f7bfdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:35.389 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.412+0000 7f7bfdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:35.389 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.412+0000 7f7bfdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:35.389 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.412+0000 7f7bfdffb640 10 monclient(hunting): my global_id is 4272 2024-03-29T11:21:35.389 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.412+0000 7f7bfdffb640 1 -- 172.21.15.86:0/2662079136 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bec004eb0 con 0x7f7c080a9ac0 2024-03-29T11:21:35.392 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.414+0000 7f7bfdffb640 1 -- 172.21.15.86:0/2662079136 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3122539179 0 0) 0x7f7bec004eb0 con 0x7f7c080a9ac0 2024-03-29T11:21:35.392 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.415+0000 7f7bfdffb640 1 -- 172.21.15.86:0/2662079136 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c08142660 con 0x7f7c080a9ac0 2024-03-29T11:21:35.392 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.415+0000 7f7bfdffb640 10 client.?.objecter ms_handle_connect 0x7f7c08071d70 2024-03-29T11:21:35.392 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.415+0000 7f7bfdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:35.392 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.415+0000 7f7bfdffb640 1 -- 172.21.15.86:0/2662079136 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f7c04001960 con 0x7f7c080a9ac0 2024-03-29T11:21:35.392 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.415+0000 7f7bfdffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:35.392 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.415+0000 7f7bfdffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:35.393 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.415+0000 7f7bfdffb640 10 monclient(hunting): dump: 2024-03-29T11:21:35.393 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:21:35.393 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:35.393 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:35.393 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:35.393 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:35.393 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:35.393 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:21:35.393 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:21:35.393 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:21:35.393 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:35.393 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.415+0000 7f7bfdffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:35.393 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.415+0000 7f7bfdffb640 1 -- 172.21.15.86:0/2662079136 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2909354906 0 0) 0x7f7c081d6cf0 con 0x7f7c08071d70 2024-03-29T11:21:35.393 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.415+0000 7f7bfdffb640 10 monclient(hunting): _negotiate 2024-03-29T11:21:35.394 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.415+0000 7f7bfdffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:35.394 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.415+0000 7f7bfdffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:35.394 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.415+0000 7f7bfdffb640 10 monclient(hunting): my global_id is 4271 2024-03-29T11:21:35.394 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.415+0000 7f7bfdffb640 1 -- 172.21.15.86:0/2662079136 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bec004eb0 con 0x7f7c08071d70 2024-03-29T11:21:35.395 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.417+0000 7f7bfdffb640 10 client.?.objecter ms_handle_connect 0x7f7c080af900 2024-03-29T11:21:35.395 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.417+0000 7f7bfdffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:35.395 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.417+0000 7f7bfdffb640 1 -- 172.21.15.86:0/2662079136 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 11248307 0 0) 0x7f7bec004eb0 con 0x7f7c08071d70 2024-03-29T11:21:35.396 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.417+0000 7f7bfdffb640 1 -- 172.21.15.86:0/2662079136 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7c081d6cf0 con 0x7f7c08071d70 2024-03-29T11:21:35.396 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.417+0000 7f7bfdffb640 1 -- 172.21.15.86:0/2662079136 <== mon.0 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 779209031 0 0) 0x7f7c08142660 con 0x7f7c080a9ac0 2024-03-29T11:21:35.396 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.417+0000 7f7bfdffb640 1 -- 172.21.15.86:0/2662079136 >> v1:172.21.15.86:6790/0 conn(0x7f7c08071d70 legacy=0x7f7c081403a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:35.396 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.417+0000 7f7bfdffb640 1 -- 172.21.15.86:0/2662079136 >> v1:172.21.15.196:6789/0 conn(0x7f7c080af900 legacy=0x7f7c081366c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:35.396 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.417+0000 7f7bfdffb640 10 monclient: _finish_hunting 0 2024-03-29T11:21:35.396 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.417+0000 7f7bfdffb640 1 monclient: found mon.a 2024-03-29T11:21:35.396 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.417+0000 7f7bfdffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:35.396 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.417+0000 7f7bfdffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:21:35.396 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.417+0000 7f7bfdffb640 1 -- 172.21.15.86:0/2662079136 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c08136dd0 con 0x7f7c080a9ac0 2024-03-29T11:21:35.396 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.417+0000 7f7bfdffb640 10 monclient: _finish_auth 0 2024-03-29T11:21:35.396 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.417+0000 7f7bfdffb640 10 monclient: _check_auth_tickets 2024-03-29T11:21:35.396 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.417+0000 7f7bfdffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:35.396 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.417+0000 7f7c0f9e2640 5 monclient: authenticate success, global_id 4272 2024-03-29T11:21:35.397 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.417+0000 7f7c0f9e2640 10 monclient: _renew_subs 2024-03-29T11:21:35.397 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.417+0000 7f7c0f9e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:21:35.397 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.417+0000 7f7c0f9e2640 1 -- 172.21.15.86:0/2662079136 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c08137080 con 0x7f7c080a9ac0 2024-03-29T11:21:35.397 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.417+0000 7f7bfdffb640 1 -- 172.21.15.86:0/2662079136 <== mon.0 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7c04001ac0 con 0x7f7c080a9ac0 2024-03-29T11:21:35.397 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.417+0000 7f7bfdffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:35.397 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.417+0000 7f7bfdffb640 1 -- 172.21.15.86:0/2662079136 <== mon.0 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f7c04003a60 con 0x7f7c080a9ac0 2024-03-29T11:21:35.397 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.418+0000 7f7bfdffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:35.397 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.418+0000 7f7bfdffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:35.397 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.418+0000 7f7bfdffb640 10 monclient: dump: 2024-03-29T11:21:35.397 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:21:35.397 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:35.397 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:35.397 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:35.398 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:35.398 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:35.398 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:21:35.398 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:21:35.398 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:21:35.398 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:35.398 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.417+0000 7f7bfeffd640 4 set_mon_vals no callback set 2024-03-29T11:21:35.398 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.418+0000 7f7c0f9e2640 10 client.4272.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:35.398 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.418+0000 7f7c0f9e2640 10 monclient: _renew_subs 2024-03-29T11:21:35.398 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.418+0000 7f7c0f9e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:21:35.398 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.418+0000 7f7c0f9e2640 1 -- 172.21.15.86:0/2662079136 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c08140c90 con 0x7f7c080a9ac0 2024-03-29T11:21:35.398 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.418+0000 7f7c0f9e2640 1 librados: init done 2024-03-29T11:21:35.400 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.420+0000 7f7bfdffb640 1 -- 172.21.15.86:0/2662079136 <== mon.0 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f7c04003bc0 con 0x7f7c080a9ac0 2024-03-29T11:21:35.400 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.420+0000 7f7bfdffb640 1 -- 172.21.15.86:0/2662079136 <== mon.0 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f7c04084a40 con 0x7f7c080a9ac0 2024-03-29T11:21:35.400 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.420+0000 7f7bfdffb640 10 client.4272.objecter ms_dispatch 0x7f7c08000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:35.400 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.420+0000 7f7bfdffb640 3 client.4272.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:35.400 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.420+0000 7f7c0cf7d640 10 monclient: get_auth_request con 0x7f7bec0721a0 auth_method 0 2024-03-29T11:21:35.400 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.421+0000 7f7bfdffb640 3 client.4272.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:35.400 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.421+0000 7f7bfdffb640 20 client.4272.objecter dump_active .. 0 homeless 2024-03-29T11:21:35.400 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.421+0000 7f7bfdffb640 10 client.4272.objecter ms_handle_connect 0x7f7bec0721a0 2024-03-29T11:21:35.400 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.421+0000 7f7c0f9e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:35.401 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.421+0000 7f7c0f9e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:35.401 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.421+0000 7f7c0f9e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:21:35.401 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.421+0000 7f7c0f9e2640 1 -- 172.21.15.86:0/2662079136 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c08075f10 con 0x7f7c080a9ac0 2024-03-29T11:21:35.407 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.430+0000 7f7bfdffb640 1 -- 172.21.15.86:0/2662079136 <== mon.0 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7c04050320 con 0x7f7c080a9ac0 2024-03-29T11:21:35.407 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.431+0000 7f7bfdffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:35.407 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.431+0000 7f7bfdffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:35.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.747+0000 7f7c0f9e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:35.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.747+0000 7f7c0f9e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:35.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.747+0000 7f7c0f9e2640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:21:35.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.747+0000 7f7c0f9e2640 1 -- 172.21.15.86:0/2662079136 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7c081d7020 con 0x7f7c080a9ac0 2024-03-29T11:21:35.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.748+0000 7f7bfdffb640 1 -- 172.21.15.86:0/2662079136 <== mon.0 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f7c04052250 con 0x7f7c080a9ac0 2024-03-29T11:21:35.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.748+0000 7f7bfdffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:35.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.748+0000 7f7bfdffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:35.726 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:21:35.726 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:35.729 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.752+0000 7f7c0f9e2640 10 librados: watch_flush enter 2024-03-29T11:21:35.729 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.753+0000 7f7c0f9e2640 10 librados: watch_flush exit 2024-03-29T11:21:35.729 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.753+0000 7f7c0f9e2640 20 client.4272.objecter shutdown clearing up homeless session... 2024-03-29T11:21:35.729 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.753+0000 7f7c0f9e2640 10 client.4272.objecter successfully canceled tick 2024-03-29T11:21:35.730 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.753+0000 7f7c0f9e2640 1 -- 172.21.15.86:0/2662079136 >> v1:172.21.15.196:6816/588110318 conn(0x7f7bec0721a0 legacy=0x7f7bec074660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:35.730 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.753+0000 7f7c0f9e2640 10 monclient: shutdown 2024-03-29T11:21:35.730 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.753+0000 7f7c0f9e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:35.730 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.753+0000 7f7c0f9e2640 1 -- 172.21.15.86:0/2662079136 >> v1:172.21.15.86:6789/0 conn(0x7f7c080a9ac0 legacy=0x7f7c081d6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:35.730 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.753+0000 7f7c0f9e2640 1 -- 172.21.15.86:0/2662079136 shutdown_connections 2024-03-29T11:21:35.730 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.753+0000 7f7c0f9e2640 1 -- 172.21.15.86:0/2662079136 >> 172.21.15.86:0/2662079136 conn(0x7f7c080b35e0 msgr2=0x7f7c08137740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:35.730 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.753+0000 7f7c0f9e2640 1 -- 172.21.15.86:0/2662079136 shutdown_connections 2024-03-29T11:21:35.730 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.753+0000 7f7c0f9e2640 1 -- 172.21.15.86:0/2662079136 wait complete. 2024-03-29T11:21:35.730 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:35.754+0000 7f7c0f9e2640 1 librados: shutdown 2024-03-29T11:21:35.749 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:42.555 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:42.710 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.732+0000 7fc2a30fb640 10 monclient: get_monmap_and_config 2024-03-29T11:21:42.710 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.732+0000 7fc2a30fb640 10 monclient: build_initial_monmap 2024-03-29T11:21:42.710 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.732+0000 7fc2a30fb640 10 monclient: monmap: 2024-03-29T11:21:42.710 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:21:42.710 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:42.710 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:21:42.734936+0000 2024-03-29T11:21:42.711 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:21:42.734936+0000 2024-03-29T11:21:42.711 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:42.711 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:42.711 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:21:42.711 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:21:42.711 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:21:42.711 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:42.712 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.734+0000 7fc2a30fb640 1 Processor -- start 2024-03-29T11:21:42.712 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.734+0000 7fc2a30fb640 1 -- start start 2024-03-29T11:21:42.712 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.734+0000 7fc2a30fb640 10 monclient: init 2024-03-29T11:21:42.713 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.735+0000 7fc2a30fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:42.713 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.735+0000 7fc2a30fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:42.713 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.735+0000 7fc2a30fb640 10 monclient(hunting): picked mon.noname-c con 0x7fc2940389e0 addr v1:172.21.15.196:6789/0 2024-03-29T11:21:42.713 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.735+0000 7fc2a30fb640 10 monclient(hunting): picked mon.noname-b con 0x7fc294038dc0 addr v1:172.21.15.86:6790/0 2024-03-29T11:21:42.713 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.735+0000 7fc2a30fb640 10 monclient(hunting): picked mon.noname-a con 0x7fc294030660 addr v1:172.21.15.86:6789/0 2024-03-29T11:21:42.713 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.735+0000 7fc2a30fb640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2940c54c0 con 0x7fc294030660 2024-03-29T11:21:42.713 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.735+0000 7fc2a30fb640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc294036430 con 0x7fc2940389e0 2024-03-29T11:21:42.713 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.735+0000 7fc2a30fb640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc294039840 con 0x7fc294038dc0 2024-03-29T11:21:42.713 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.735+0000 7fc2a30fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:42.713 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.735+0000 7fc2a30fb640 10 monclient(hunting): authenticate will time out at 927.054785s 2024-03-29T11:21:42.719 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.742+0000 7fc2a28fa640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc294030660 0x7fc2940d3170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:39795/0 (socket says 172.21.15.86:39795) 2024-03-29T11:21:42.719 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.742+0000 7fc2a28fa640 1 -- 172.21.15.86:0/58147604 learned_addr learned my addr 172.21.15.86:0/58147604 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:21:42.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.742+0000 7fc2a20f9640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fc2940389e0 0x7fc294033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:39295/0 (socket says 172.21.15.86:39295) 2024-03-29T11:21:42.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.742+0000 7fc2a10f7640 1 -- 172.21.15.86:0/58147604 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2943387614 0 0) 0x7fc2940c54c0 con 0x7fc294030660 2024-03-29T11:21:42.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.742+0000 7fc2a10f7640 10 monclient(hunting): _negotiate 2024-03-29T11:21:42.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.742+0000 7fc2a10f7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:42.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.742+0000 7fc2a10f7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:42.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.742+0000 7fc2a10f7640 10 monclient(hunting): my global_id is 4275 2024-03-29T11:21:42.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.742+0000 7fc2a10f7640 1 -- 172.21.15.86:0/58147604 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc288003520 con 0x7fc294030660 2024-03-29T11:21:42.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.743+0000 7fc2a10f7640 1 -- 172.21.15.86:0/58147604 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2912689500 0 0) 0x7fc294036430 con 0x7fc2940389e0 2024-03-29T11:21:42.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.743+0000 7fc2a10f7640 10 monclient(hunting): _negotiate 2024-03-29T11:21:42.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.743+0000 7fc2a10f7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:42.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.743+0000 7fc2a10f7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:42.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.743+0000 7fc2a10f7640 10 monclient(hunting): my global_id is 4255 2024-03-29T11:21:42.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.743+0000 7fc2a10f7640 1 -- 172.21.15.86:0/58147604 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc2940c54c0 con 0x7fc2940389e0 2024-03-29T11:21:42.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.747+0000 7fc2a10f7640 1 -- 172.21.15.86:0/58147604 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3388780400 0 0) 0x7fc288003520 con 0x7fc294030660 2024-03-29T11:21:42.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.747+0000 7fc2a10f7640 1 -- 172.21.15.86:0/58147604 >> v1:172.21.15.86:6790/0 conn(0x7fc294038dc0 legacy=0x7fc2940cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:42.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.747+0000 7fc2a10f7640 1 -- 172.21.15.86:0/58147604 >> v1:172.21.15.196:6789/0 conn(0x7fc2940389e0 legacy=0x7fc294033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:42.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.747+0000 7fc2a10f7640 10 monclient: _finish_hunting 0 2024-03-29T11:21:42.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.747+0000 7fc2a10f7640 1 monclient: found mon.noname-a 2024-03-29T11:21:42.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.747+0000 7fc2a10f7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:42.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.747+0000 7fc2a10f7640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.86:6789/0 2024-03-29T11:21:42.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.747+0000 7fc2a10f7640 1 -- 172.21.15.86:0/58147604 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc294035690 con 0x7fc294030660 2024-03-29T11:21:42.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.747+0000 7fc2a10f7640 10 monclient: _finish_auth 0 2024-03-29T11:21:42.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.747+0000 7fc2a10f7640 10 monclient: _check_auth_tickets 2024-03-29T11:21:42.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.747+0000 7fc2a10f7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:42.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.747+0000 7fc2a30fb640 5 monclient: authenticate success, global_id 4275 2024-03-29T11:21:42.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.747+0000 7fc2a30fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:42.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.747+0000 7fc2a10f7640 1 -- 172.21.15.86:0/58147604 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc29c04f730 con 0x7fc294030660 2024-03-29T11:21:42.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.747+0000 7fc2a10f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:42.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.747+0000 7fc2a10f7640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:21:42.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.747+0000 7fc2a10f7640 10 monclient: dump: 2024-03-29T11:21:42.725 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:21:42.725 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:42.725 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:42.726 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:42.726 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:42.726 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:42.726 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:21:42.726 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:21:42.726 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:21:42.726 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:42.726 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.747+0000 7fc2a30fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:42.727 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.748+0000 7fc2a10f7640 1 -- 172.21.15.86:0/58147604 <== mon.0 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc29c0008d0 con 0x7fc294030660 2024-03-29T11:21:42.727 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.748+0000 7fc2a10f7640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:42.728 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.748+0000 7fc2a30fb640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:42.728 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.748+0000 7fc2a30fb640 4 set_mon_vals no callback set 2024-03-29T11:21:42.728 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.749+0000 7fc2a10f7640 1 -- 172.21.15.86:0/58147604 <== mon.0 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc29c0758b0 con 0x7fc294030660 2024-03-29T11:21:42.728 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.749+0000 7fc2a10f7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:42.728 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.749+0000 7fc2a10f7640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:21:42.728 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.749+0000 7fc2a10f7640 10 monclient: dump: 2024-03-29T11:21:42.728 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:21:42.728 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:42.728 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:42.728 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:42.728 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:42.728 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:42.728 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:21:42.728 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:21:42.729 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:21:42.729 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:42.729 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.749+0000 7fc2a30fb640 10 monclient: shutdown 2024-03-29T11:21:42.729 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.749+0000 7fc2a30fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:42.729 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.749+0000 7fc2a30fb640 1 -- 172.21.15.86:0/58147604 >> v1:172.21.15.86:6789/0 conn(0x7fc294030660 legacy=0x7fc2940d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:42.729 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.749+0000 7fc2a30fb640 1 -- 172.21.15.86:0/58147604 shutdown_connections 2024-03-29T11:21:42.729 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.749+0000 7fc2a30fb640 1 -- 172.21.15.86:0/58147604 >> 172.21.15.86:0/58147604 conn(0x7fc294005c00 msgr2=0x7fc294006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:42.730 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.752+0000 7fc2a30fb640 1 -- 172.21.15.86:0/58147604 shutdown_connections 2024-03-29T11:21:42.730 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.752+0000 7fc2a30fb640 1 -- 172.21.15.86:0/58147604 wait complete. 2024-03-29T11:21:42.732 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.753+0000 7fc2a30fb640 10 monclient: build_initial_monmap 2024-03-29T11:21:42.732 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.753+0000 7fc2a30fb640 10 monclient: monmap: 2024-03-29T11:21:42.732 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:21:42.732 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:42.732 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:21:42.732 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:21:42.732 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:42.732 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:42.732 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:21:42.732 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:21:42.732 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:21:42.733 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:42.733 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.754+0000 7fc2a30fb640 1 librados: starting msgr at 2024-03-29T11:21:42.733 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.754+0000 7fc2a30fb640 1 librados: starting objecter 2024-03-29T11:21:42.733 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.754+0000 7fc2a30fb640 1 Processor -- start 2024-03-29T11:21:42.733 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.754+0000 7fc2a30fb640 1 -- start start 2024-03-29T11:21:42.733 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.754+0000 7fc2a30fb640 1 librados: setting wanted keys 2024-03-29T11:21:42.733 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.754+0000 7fc2a30fb640 1 librados: calling monclient init 2024-03-29T11:21:42.733 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.754+0000 7fc2a30fb640 10 monclient: init 2024-03-29T11:21:42.733 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.755+0000 7fc2a30fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:42.735 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.755+0000 7fc2a30fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:42.735 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.755+0000 7fc2a30fb640 10 monclient(hunting): picked mon.noname-b con 0x7fc294030660 addr v1:172.21.15.196:6789/0 2024-03-29T11:21:42.735 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.755+0000 7fc2a30fb640 10 monclient(hunting): picked mon.noname-c con 0x7fc2940389e0 addr v1:172.21.15.86:6790/0 2024-03-29T11:21:42.735 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.755+0000 7fc2a30fb640 10 monclient(hunting): picked mon.noname-a con 0x7fc294161710 addr v1:172.21.15.86:6789/0 2024-03-29T11:21:42.736 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.755+0000 7fc2a30fb640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc29402f440 con 0x7fc294161710 2024-03-29T11:21:42.736 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.755+0000 7fc2a30fb640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc29402f5d0 con 0x7fc294030660 2024-03-29T11:21:42.736 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.755+0000 7fc2a30fb640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc29402f760 con 0x7fc2940389e0 2024-03-29T11:21:42.736 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.755+0000 7fc2a30fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:42.736 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.755+0000 7fc2a30fb640 10 monclient(hunting): authenticate will time out at 927.074620s 2024-03-29T11:21:42.737 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.760+0000 7fc2a28fa640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc294161710 0x7fc2940c5c10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:39957/0 (socket says 172.21.15.86:39957) 2024-03-29T11:21:42.737 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.760+0000 7fc2a28fa640 1 -- 172.21.15.86:0/547267545 learned_addr learned my addr 172.21.15.86:0/547267545 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:21:42.740 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.763+0000 7fc292ffd640 10 client.?.objecter ms_handle_connect 0x7fc294161710 2024-03-29T11:21:42.740 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.763+0000 7fc292ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:42.745 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.764+0000 7fc292ffd640 10 client.?.objecter ms_handle_connect 0x7fc2940389e0 2024-03-29T11:21:42.745 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.764+0000 7fc292ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:42.745 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.764+0000 7fc292ffd640 1 -- 172.21.15.86:0/547267545 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 362499058 0 0) 0x7fc29402f760 con 0x7fc2940389e0 2024-03-29T11:21:42.745 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.764+0000 7fc292ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:42.745 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.764+0000 7fc292ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:42.745 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.764+0000 7fc292ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:42.745 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.764+0000 7fc292ffd640 10 monclient(hunting): my global_id is 4277 2024-03-29T11:21:42.745 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.764+0000 7fc292ffd640 1 -- 172.21.15.86:0/547267545 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc280004eb0 con 0x7fc2940389e0 2024-03-29T11:21:42.745 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.764+0000 7fc292ffd640 1 -- 172.21.15.86:0/547267545 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3459751133 0 0) 0x7fc29402f440 con 0x7fc294161710 2024-03-29T11:21:42.745 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.764+0000 7fc292ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:42.745 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.764+0000 7fc292ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:42.745 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.764+0000 7fc292ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:42.745 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.764+0000 7fc292ffd640 10 monclient(hunting): my global_id is 4278 2024-03-29T11:21:42.745 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.764+0000 7fc292ffd640 1 -- 172.21.15.86:0/547267545 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc29402f760 con 0x7fc294161710 2024-03-29T11:21:42.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.764+0000 7fc292ffd640 1 -- 172.21.15.86:0/547267545 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 857438560 0 0) 0x7fc280004eb0 con 0x7fc2940389e0 2024-03-29T11:21:42.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.764+0000 7fc292ffd640 1 -- 172.21.15.86:0/547267545 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc29402f440 con 0x7fc2940389e0 2024-03-29T11:21:42.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.765+0000 7fc292ffd640 1 -- 172.21.15.86:0/547267545 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc28c001850 con 0x7fc2940389e0 2024-03-29T11:21:42.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.765+0000 7fc292ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:42.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.765+0000 7fc292ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:42.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.765+0000 7fc292ffd640 10 monclient(hunting): dump: 2024-03-29T11:21:42.746 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:21:42.746 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:42.746 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:42.746 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:42.746 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:42.746 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:42.746 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:21:42.746 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:21:42.747 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:21:42.747 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:42.747 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.765+0000 7fc292ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:42.747 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.765+0000 7fc292ffd640 1 -- 172.21.15.86:0/547267545 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3026041726 0 0) 0x7fc29402f440 con 0x7fc2940389e0 2024-03-29T11:21:42.747 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.765+0000 7fc292ffd640 1 -- 172.21.15.86:0/547267545 >> v1:172.21.15.196:6789/0 conn(0x7fc294030660 legacy=0x7fc294161000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:42.747 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.765+0000 7fc292ffd640 1 -- 172.21.15.86:0/547267545 >> v1:172.21.15.86:6789/0 conn(0x7fc294161710 legacy=0x7fc2940c5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:42.747 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.765+0000 7fc292ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:42.747 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.765+0000 7fc292ffd640 1 monclient: found mon.c 2024-03-29T11:21:42.747 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.766+0000 7fc292ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:42.747 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.766+0000 7fc292ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:21:42.747 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.766+0000 7fc292ffd640 1 -- 172.21.15.86:0/547267545 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2940c3050 con 0x7fc2940389e0 2024-03-29T11:21:42.747 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.766+0000 7fc292ffd640 10 monclient: _finish_auth 0 2024-03-29T11:21:42.747 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.766+0000 7fc292ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:42.747 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.766+0000 7fc292ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:42.747 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.766+0000 7fc2a30fb640 5 monclient: authenticate success, global_id 4277 2024-03-29T11:21:42.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.766+0000 7fc2a30fb640 10 monclient: _renew_subs 2024-03-29T11:21:42.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.766+0000 7fc2a30fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:21:42.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.766+0000 7fc2a30fb640 1 -- 172.21.15.86:0/547267545 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2940c32a0 con 0x7fc2940389e0 2024-03-29T11:21:42.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.766+0000 7fc2a30fb640 10 client.4277.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:42.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.766+0000 7fc2a30fb640 10 monclient: _renew_subs 2024-03-29T11:21:42.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.766+0000 7fc2a30fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:21:42.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.766+0000 7fc2a30fb640 1 -- 172.21.15.86:0/547267545 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2940c6500 con 0x7fc2940389e0 2024-03-29T11:21:42.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.766+0000 7fc2a30fb640 1 librados: init done 2024-03-29T11:21:42.749 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.768+0000 7fc2a30fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:42.749 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.768+0000 7fc2a30fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:42.749 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.768+0000 7fc2a30fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:21:42.749 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.768+0000 7fc2a30fb640 1 -- 172.21.15.86:0/547267545 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc29400a770 con 0x7fc2940389e0 2024-03-29T11:21:42.750 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.768+0000 7fc292ffd640 1 -- 172.21.15.86:0/547267545 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc28c0019b0 con 0x7fc2940389e0 2024-03-29T11:21:42.750 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.768+0000 7fc292ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:42.750 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.768+0000 7fc292ffd640 1 -- 172.21.15.86:0/547267545 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc28c003870 con 0x7fc2940389e0 2024-03-29T11:21:42.750 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.768+0000 7fc292ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:42.750 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.768+0000 7fc292ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:42.750 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.768+0000 7fc292ffd640 10 monclient: dump: 2024-03-29T11:21:42.750 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:21:42.750 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:42.750 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:42.750 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:42.750 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:42.750 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:42.750 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:21:42.750 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:21:42.751 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:21:42.751 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:42.751 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.768+0000 7fc293fff640 4 set_mon_vals no callback set 2024-03-29T11:21:42.752 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.771+0000 7fc292ffd640 1 -- 172.21.15.86:0/547267545 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7fc28c019900 con 0x7fc2940389e0 2024-03-29T11:21:42.752 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.772+0000 7fc292ffd640 1 -- 172.21.15.86:0/547267545 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7fc28c0838e0 con 0x7fc2940389e0 2024-03-29T11:21:42.752 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.772+0000 7fc292ffd640 10 client.4277.objecter ms_dispatch 0x7fc29c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:42.752 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.772+0000 7fc292ffd640 3 client.4277.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:42.752 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.772+0000 7fc292ffd640 3 client.4277.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:42.752 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.772+0000 7fc292ffd640 20 client.4277.objecter dump_active .. 0 homeless 2024-03-29T11:21:42.752 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.773+0000 7fc2a20f9640 10 monclient: get_auth_request con 0x7fc28006db70 auth_method 0 2024-03-29T11:21:42.754 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.777+0000 7fc292ffd640 10 client.4277.objecter ms_handle_connect 0x7fc28006db70 2024-03-29T11:21:42.757 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.779+0000 7fc292ffd640 1 -- 172.21.15.86:0/547267545 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc29400a770 con 0x7fc2940389e0 2024-03-29T11:21:42.757 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.779+0000 7fc292ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:42.757 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:42.779+0000 7fc292ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:43.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:43.093+0000 7fc2a30fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:43.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:43.093+0000 7fc2a30fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:43.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:43.093+0000 7fc2a30fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:21:43.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:43.093+0000 7fc2a30fb640 1 -- 172.21.15.86:0/547267545 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc29400a770 con 0x7fc2940389e0 2024-03-29T11:21:43.076 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:43.098+0000 7fc292ffd640 1 -- 172.21.15.86:0/547267545 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7fc29400a770 con 0x7fc2940389e0 2024-03-29T11:21:43.076 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:43.099+0000 7fc292ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:43.076 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:43.099+0000 7fc292ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:43.077 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:21:43.077 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:43.079 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:43.102+0000 7fc2a30fb640 10 librados: watch_flush enter 2024-03-29T11:21:43.080 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:43.103+0000 7fc2a30fb640 10 librados: watch_flush exit 2024-03-29T11:21:43.080 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:43.103+0000 7fc2a30fb640 20 client.4277.objecter shutdown clearing up homeless session... 2024-03-29T11:21:43.080 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:43.103+0000 7fc2a30fb640 10 client.4277.objecter successfully canceled tick 2024-03-29T11:21:43.082 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:43.104+0000 7fc2a30fb640 1 -- 172.21.15.86:0/547267545 >> v1:172.21.15.196:6816/588110318 conn(0x7fc28006db70 legacy=0x7fc280070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:43.082 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:43.105+0000 7fc2a30fb640 10 monclient: shutdown 2024-03-29T11:21:43.082 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:43.105+0000 7fc2a30fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:43.082 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:43.105+0000 7fc2a30fb640 1 -- 172.21.15.86:0/547267545 >> v1:172.21.15.86:6790/0 conn(0x7fc2940389e0 legacy=0x7fc2940c2940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:43.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:43.110+0000 7fc2a30fb640 1 -- 172.21.15.86:0/547267545 shutdown_connections 2024-03-29T11:21:43.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:43.110+0000 7fc2a30fb640 1 -- 172.21.15.86:0/547267545 >> 172.21.15.86:0/547267545 conn(0x7fc294005c00 msgr2=0x7fc294034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:43.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:43.110+0000 7fc2a30fb640 1 -- 172.21.15.86:0/547267545 shutdown_connections 2024-03-29T11:21:43.091 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:43.110+0000 7fc2a30fb640 1 -- 172.21.15.86:0/547267545 wait complete. 2024-03-29T11:21:43.091 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:43.111+0000 7fc2a30fb640 1 librados: shutdown 2024-03-29T11:21:43.107 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:49.915 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:50.065 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.089+0000 7fb1a8bc6640 10 monclient: get_monmap_and_config 2024-03-29T11:21:50.065 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.089+0000 7fb1a8bc6640 10 monclient: build_initial_monmap 2024-03-29T11:21:50.066 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.089+0000 7fb1a8bc6640 10 monclient: monmap: 2024-03-29T11:21:50.066 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:21:50.066 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:50.066 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:21:50.090085+0000 2024-03-29T11:21:50.066 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:21:50.090085+0000 2024-03-29T11:21:50.066 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:50.066 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:50.066 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:21:50.066 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:21:50.066 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:21:50.066 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:50.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.090+0000 7fb1a8bc6640 1 Processor -- start 2024-03-29T11:21:50.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.090+0000 7fb1a8bc6640 1 -- start start 2024-03-29T11:21:50.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.090+0000 7fb1a8bc6640 10 monclient: init 2024-03-29T11:21:50.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.091+0000 7fb1a8bc6640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:50.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.091+0000 7fb1a8bc6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:21:50.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.091+0000 7fb1a8bc6640 10 monclient(hunting): picked mon.noname-a con 0x7fb1a40b4920 addr v1:172.21.15.86:6789/0 2024-03-29T11:21:50.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.091+0000 7fb1a8bc6640 10 monclient(hunting): picked mon.noname-b con 0x7fb1a40b1790 addr v1:172.21.15.86:6790/0 2024-03-29T11:21:50.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.091+0000 7fb1a8bc6640 10 monclient(hunting): picked mon.noname-c con 0x7fb1a4146790 addr v1:172.21.15.196:6789/0 2024-03-29T11:21:50.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.091+0000 7fb1a8bc6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a40b4070 con 0x7fb1a40b4920 2024-03-29T11:21:50.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.091+0000 7fb1a8bc6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a406fb80 con 0x7fb1a4146790 2024-03-29T11:21:50.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.091+0000 7fb1a8bc6640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a40b42c0 con 0x7fb1a40b1790 2024-03-29T11:21:50.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.091+0000 7fb1a8bc6640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:50.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.091+0000 7fb1a8bc6640 10 monclient(hunting): authenticate will time out at 934.409243s 2024-03-29T11:21:50.072 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.091+0000 7fb1a2d9d640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fb1a4146790 0x7fb1a4146c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:58030/0 (socket says 172.21.15.86:58030) 2024-03-29T11:21:50.072 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.091+0000 7fb1a2d9d640 1 -- 172.21.15.86:0/422480370 learned_addr learned my addr 172.21.15.86:0/422480370 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:21:50.072 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.092+0000 7fb1a159a640 1 -- 172.21.15.86:0/422480370 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3498920511 0 0) 0x7fb1a406fb80 con 0x7fb1a4146790 2024-03-29T11:21:50.072 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.092+0000 7fb1a159a640 10 monclient(hunting): _negotiate 2024-03-29T11:21:50.072 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.092+0000 7fb1a159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:50.072 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.092+0000 7fb1a159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:50.072 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.092+0000 7fb1a159a640 10 monclient(hunting): my global_id is 4258 2024-03-29T11:21:50.072 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.092+0000 7fb1a159a640 1 -- 172.21.15.86:0/422480370 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb190003520 con 0x7fb1a4146790 2024-03-29T11:21:50.072 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.092+0000 7fb1a159a640 1 -- 172.21.15.86:0/422480370 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2615866345 0 0) 0x7fb190003520 con 0x7fb1a4146790 2024-03-29T11:21:50.072 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.092+0000 7fb1a159a640 1 -- 172.21.15.86:0/422480370 >> v1:172.21.15.86:6790/0 conn(0x7fb1a40b1790 legacy=0x7fb1a40b1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:21:50.072 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.092+0000 7fb1a159a640 1 -- 172.21.15.86:0/422480370 >> v1:172.21.15.86:6789/0 conn(0x7fb1a40b4920 legacy=0x7fb1a406e510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:50.072 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.092+0000 7fb1a159a640 10 monclient: _finish_hunting 0 2024-03-29T11:21:50.072 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.092+0000 7fb1a159a640 1 monclient: found mon.noname-c 2024-03-29T11:21:50.072 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.092+0000 7fb1a159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:50.072 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.092+0000 7fb1a159a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T11:21:50.073 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.092+0000 7fb1a159a640 1 -- 172.21.15.86:0/422480370 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1a40b22c0 con 0x7fb1a4146790 2024-03-29T11:21:50.073 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.092+0000 7fb1a159a640 10 monclient: _finish_auth 0 2024-03-29T11:21:50.073 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.092+0000 7fb1a159a640 10 monclient: _check_auth_tickets 2024-03-29T11:21:50.073 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.092+0000 7fb1a159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:50.073 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.092+0000 7fb1a159a640 1 -- 172.21.15.86:0/422480370 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fb19c001690 con 0x7fb1a4146790 2024-03-29T11:21:50.073 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.092+0000 7fb1a159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:50.073 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.092+0000 7fb1a159a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:50.073 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.092+0000 7fb1a159a640 10 monclient: dump: 2024-03-29T11:21:50.073 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:21:50.073 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:50.073 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:50.073 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:50.073 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:50.073 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:50.073 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:21:50.074 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:21:50.074 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:21:50.074 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:50.074 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.092+0000 7fb1a8bc6640 5 monclient: authenticate success, global_id 4258 2024-03-29T11:21:50.074 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.092+0000 7fb1a8bc6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:50.074 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.093+0000 7fb1a159a640 1 -- 172.21.15.86:0/422480370 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb19c001860 con 0x7fb1a4146790 2024-03-29T11:21:50.074 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.093+0000 7fb1a159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:50.074 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.093+0000 7fb1a159a640 1 -- 172.21.15.86:0/422480370 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fb19c001cb0 con 0x7fb1a4146790 2024-03-29T11:21:50.074 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.093+0000 7fb1a159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:50.074 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.093+0000 7fb1a159a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:50.074 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.093+0000 7fb1a159a640 10 monclient: dump: 2024-03-29T11:21:50.074 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:21:50.074 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:50.074 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:50.074 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:50.075 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:50.075 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:50.075 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:21:50.075 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:21:50.075 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:21:50.075 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:50.075 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.093+0000 7fb1a8bc6640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:50.075 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.093+0000 7fb1a8bc6640 4 set_mon_vals no callback set 2024-03-29T11:21:50.075 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.094+0000 7fb1a8bc6640 10 monclient: shutdown 2024-03-29T11:21:50.075 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.094+0000 7fb1a8bc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:50.075 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.094+0000 7fb1a8bc6640 1 -- 172.21.15.86:0/422480370 >> v1:172.21.15.196:6789/0 conn(0x7fb1a4146790 legacy=0x7fb1a4146c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:50.075 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.094+0000 7fb1a8bc6640 1 -- 172.21.15.86:0/422480370 shutdown_connections 2024-03-29T11:21:50.075 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.094+0000 7fb1a8bc6640 1 -- 172.21.15.86:0/422480370 >> 172.21.15.86:0/422480370 conn(0x7fb1a4066a00 msgr2=0x7fb1a4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:50.075 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.094+0000 7fb1a8bc6640 1 -- 172.21.15.86:0/422480370 shutdown_connections 2024-03-29T11:21:50.082 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.095+0000 7fb1a8bc6640 1 -- 172.21.15.86:0/422480370 wait complete. 2024-03-29T11:21:50.082 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.096+0000 7fb1a8bc6640 10 monclient: build_initial_monmap 2024-03-29T11:21:50.082 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.096+0000 7fb1a8bc6640 10 monclient: monmap: 2024-03-29T11:21:50.082 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:21:50.082 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:50.082 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:21:50.082 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:21:50.083 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:50.083 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:50.083 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:21:50.083 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:21:50.083 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:21:50.083 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:50.083 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.096+0000 7fb1a8bc6640 1 librados: starting msgr at 2024-03-29T11:21:50.083 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.096+0000 7fb1a8bc6640 1 librados: starting objecter 2024-03-29T11:21:50.083 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.096+0000 7fb1a8bc6640 1 Processor -- start 2024-03-29T11:21:50.083 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.096+0000 7fb1a8bc6640 1 -- start start 2024-03-29T11:21:50.083 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.096+0000 7fb1a8bc6640 1 librados: setting wanted keys 2024-03-29T11:21:50.083 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.096+0000 7fb1a8bc6640 1 librados: calling monclient init 2024-03-29T11:21:50.083 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.096+0000 7fb1a8bc6640 10 monclient: init 2024-03-29T11:21:50.083 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.097+0000 7fb1a8bc6640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:50.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.097+0000 7fb1a8bc6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:21:50.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.097+0000 7fb1a8bc6640 10 monclient(hunting): picked mon.noname-c con 0x7fb1a40b1790 addr v1:172.21.15.86:6790/0 2024-03-29T11:21:50.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.097+0000 7fb1a8bc6640 10 monclient(hunting): picked mon.noname-a con 0x7fb1a40b4920 addr v1:172.21.15.86:6789/0 2024-03-29T11:21:50.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.097+0000 7fb1a8bc6640 10 monclient(hunting): picked mon.noname-b con 0x7fb1a4146790 addr v1:172.21.15.196:6789/0 2024-03-29T11:21:50.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.097+0000 7fb1a8bc6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a41376f0 con 0x7fb1a40b4920 2024-03-29T11:21:50.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.097+0000 7fb1a8bc6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a4136d30 con 0x7fb1a4146790 2024-03-29T11:21:50.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.097+0000 7fb1a8bc6640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1a4137920 con 0x7fb1a40b1790 2024-03-29T11:21:50.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.097+0000 7fb1a8bc6640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:50.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.097+0000 7fb1a8bc6640 10 monclient(hunting): authenticate will time out at 934.415618s 2024-03-29T11:21:50.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.097+0000 7fb1a2d9d640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fb1a4146790 0x7fb1a41453f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:58292/0 (socket says 172.21.15.86:58292) 2024-03-29T11:21:50.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.097+0000 7fb1a2d9d640 1 -- 172.21.15.86:0/2876171094 learned_addr learned my addr 172.21.15.86:0/2876171094 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:21:50.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.097+0000 7fb19affd640 10 client.?.objecter ms_handle_connect 0x7fb1a4146790 2024-03-29T11:21:50.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.097+0000 7fb19affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:50.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.098+0000 7fb19affd640 1 -- 172.21.15.86:0/2876171094 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2638365055 0 0) 0x7fb1a4136d30 con 0x7fb1a4146790 2024-03-29T11:21:50.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.098+0000 7fb19affd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:50.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.098+0000 7fb19affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:50.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.098+0000 7fb19affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:50.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.098+0000 7fb19affd640 10 monclient(hunting): my global_id is 4261 2024-03-29T11:21:50.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.098+0000 7fb19affd640 1 -- 172.21.15.86:0/2876171094 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb184004eb0 con 0x7fb1a4146790 2024-03-29T11:21:50.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.098+0000 7fb19affd640 10 client.?.objecter ms_handle_connect 0x7fb1a40b1790 2024-03-29T11:21:50.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.098+0000 7fb19affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:50.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.098+0000 7fb19affd640 1 -- 172.21.15.86:0/2876171094 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3664087139 0 0) 0x7fb184004eb0 con 0x7fb1a4146790 2024-03-29T11:21:50.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.098+0000 7fb19affd640 1 -- 172.21.15.86:0/2876171094 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fb1a4136d30 con 0x7fb1a4146790 2024-03-29T11:21:50.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.098+0000 7fb19affd640 1 -- 172.21.15.86:0/2876171094 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fb19c001b70 con 0x7fb1a4146790 2024-03-29T11:21:50.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.098+0000 7fb19affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:50.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.098+0000 7fb19affd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:50.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.098+0000 7fb19affd640 10 monclient(hunting): dump: 2024-03-29T11:21:50.085 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:21:50.085 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:50.086 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:50.086 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:50.086 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:50.086 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:50.086 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:21:50.086 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:21:50.086 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:21:50.086 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:50.086 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.098+0000 7fb19affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:50.086 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.098+0000 7fb19affd640 1 -- 172.21.15.86:0/2876171094 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2172136638 0 0) 0x7fb1a4137920 con 0x7fb1a40b1790 2024-03-29T11:21:50.086 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.098+0000 7fb19affd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:50.086 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.098+0000 7fb19affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:50.086 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.098+0000 7fb19affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:50.086 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.098+0000 7fb19affd640 10 monclient(hunting): my global_id is 4280 2024-03-29T11:21:50.086 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.098+0000 7fb19affd640 1 -- 172.21.15.86:0/2876171094 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb184004eb0 con 0x7fb1a40b1790 2024-03-29T11:21:50.087 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.098+0000 7fb19affd640 10 client.?.objecter ms_handle_connect 0x7fb1a40b4920 2024-03-29T11:21:50.087 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.098+0000 7fb19affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:50.087 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.099+0000 7fb19affd640 1 -- 172.21.15.86:0/2876171094 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2079963998 0 0) 0x7fb1a4136d30 con 0x7fb1a4146790 2024-03-29T11:21:50.087 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.099+0000 7fb19affd640 1 -- 172.21.15.86:0/2876171094 >> v1:172.21.15.86:6790/0 conn(0x7fb1a40b1790 legacy=0x7fb1a4138300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:50.087 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.099+0000 7fb19affd640 1 -- 172.21.15.86:0/2876171094 >> v1:172.21.15.86:6789/0 conn(0x7fb1a40b4920 legacy=0x7fb1a4136620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:50.087 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.099+0000 7fb19affd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:50.087 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.099+0000 7fb19affd640 1 monclient: found mon.b 2024-03-29T11:21:50.087 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.099+0000 7fb19affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:50.087 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.099+0000 7fb19affd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:21:50.087 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.099+0000 7fb19affd640 1 -- 172.21.15.86:0/2876171094 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1a4137c00 con 0x7fb1a4146790 2024-03-29T11:21:50.087 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.099+0000 7fb19affd640 10 monclient: _finish_auth 0 2024-03-29T11:21:50.087 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.099+0000 7fb19affd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:50.087 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.099+0000 7fb19affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:50.087 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.099+0000 7fb1a8bc6640 5 monclient: authenticate success, global_id 4261 2024-03-29T11:21:50.088 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.099+0000 7fb1a8bc6640 10 monclient: _renew_subs 2024-03-29T11:21:50.088 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.099+0000 7fb1a8bc6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:21:50.088 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.099+0000 7fb1a8bc6640 1 -- 172.21.15.86:0/2876171094 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1a4137ee0 con 0x7fb1a4146790 2024-03-29T11:21:50.088 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.099+0000 7fb1a8bc6640 10 client.4261.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:50.088 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.099+0000 7fb1a8bc6640 10 monclient: _renew_subs 2024-03-29T11:21:50.088 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.099+0000 7fb1a8bc6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:21:50.088 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.099+0000 7fb1a8bc6640 1 -- 172.21.15.86:0/2876171094 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1a41e7310 con 0x7fb1a4146790 2024-03-29T11:21:50.088 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.099+0000 7fb1a8bc6640 1 librados: init done 2024-03-29T11:21:50.088 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.099+0000 7fb19affd640 1 -- 172.21.15.86:0/2876171094 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fb19c004a90 con 0x7fb1a4146790 2024-03-29T11:21:50.088 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.099+0000 7fb19affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:50.088 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.099+0000 7fb19affd640 1 -- 172.21.15.86:0/2876171094 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fb19c004da0 con 0x7fb1a4146790 2024-03-29T11:21:50.088 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.099+0000 7fb19affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:50.088 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.099+0000 7fb19affd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:21:50.088 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.099+0000 7fb19affd640 10 monclient: dump: 2024-03-29T11:21:50.088 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:21:50.089 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:50.089 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:50.089 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:50.089 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:50.089 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:50.089 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:21:50.089 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:21:50.089 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:21:50.089 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:50.089 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.099+0000 7fb1a0d99640 4 set_mon_vals no callback set 2024-03-29T11:21:50.089 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.100+0000 7fb1a8bc6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:50.089 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.101+0000 7fb1a8bc6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:50.089 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.101+0000 7fb1a8bc6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:21:50.089 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.101+0000 7fb1a8bc6640 1 -- 172.21.15.86:0/2876171094 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb188005080 con 0x7fb1a4146790 2024-03-29T11:21:50.089 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.102+0000 7fb19affd640 1 -- 172.21.15.86:0/2876171094 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7fb19c01ad70 con 0x7fb1a4146790 2024-03-29T11:21:50.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.103+0000 7fb19affd640 1 -- 172.21.15.86:0/2876171094 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7fb19c084aa0 con 0x7fb1a4146790 2024-03-29T11:21:50.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.103+0000 7fb19affd640 10 client.4261.objecter ms_dispatch 0x7fb1a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:50.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.103+0000 7fb19affd640 3 client.4261.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:50.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.103+0000 7fb19affd640 3 client.4261.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:50.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.103+0000 7fb19affd640 20 client.4261.objecter dump_active .. 0 homeless 2024-03-29T11:21:50.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.103+0000 7fb1a259c640 10 monclient: get_auth_request con 0x7fb18406ddc0 auth_method 0 2024-03-29T11:21:50.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.104+0000 7fb19affd640 10 client.4261.objecter ms_handle_connect 0x7fb18406ddc0 2024-03-29T11:21:50.091 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.111+0000 7fb19affd640 1 -- 172.21.15.86:0/2876171094 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fb19c050380 con 0x7fb1a4146790 2024-03-29T11:21:50.091 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.111+0000 7fb19affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:50.091 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.111+0000 7fb19affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:50.492 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.515+0000 7fb1a8bc6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:50.492 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.515+0000 7fb1a8bc6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:50.492 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.515+0000 7fb1a8bc6640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:21:50.492 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.515+0000 7fb1a8bc6640 1 -- 172.21.15.86:0/2876171094 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb188005330 con 0x7fb1a4146790 2024-03-29T11:21:50.493 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:21:50.493 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:50.493 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.516+0000 7fb19affd640 1 -- 172.21.15.86:0/2876171094 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7fb19c0522b0 con 0x7fb1a4146790 2024-03-29T11:21:50.493 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.516+0000 7fb19affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:50.494 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.516+0000 7fb19affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:50.497 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.519+0000 7fb1a8bc6640 10 librados: watch_flush enter 2024-03-29T11:21:50.497 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.520+0000 7fb1a8bc6640 10 librados: watch_flush exit 2024-03-29T11:21:50.497 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.520+0000 7fb1a8bc6640 20 client.4261.objecter shutdown clearing up homeless session... 2024-03-29T11:21:50.497 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.520+0000 7fb1a8bc6640 10 client.4261.objecter successfully canceled tick 2024-03-29T11:21:50.497 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.520+0000 7fb1a8bc6640 1 -- 172.21.15.86:0/2876171094 >> v1:172.21.15.196:6816/588110318 conn(0x7fb18406ddc0 legacy=0x7fb184070280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:50.497 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.520+0000 7fb1a8bc6640 10 monclient: shutdown 2024-03-29T11:21:50.497 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.520+0000 7fb1a8bc6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:50.497 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.520+0000 7fb1a8bc6640 1 -- 172.21.15.86:0/2876171094 >> v1:172.21.15.196:6789/0 conn(0x7fb1a4146790 legacy=0x7fb1a41453f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:50.497 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.520+0000 7fb1a8bc6640 1 -- 172.21.15.86:0/2876171094 shutdown_connections 2024-03-29T11:21:50.497 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.520+0000 7fb1a8bc6640 1 -- 172.21.15.86:0/2876171094 >> 172.21.15.86:0/2876171094 conn(0x7fb1a4066a00 msgr2=0x7fb1a406ff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:50.497 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.520+0000 7fb1a8bc6640 1 -- 172.21.15.86:0/2876171094 shutdown_connections 2024-03-29T11:21:50.497 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.520+0000 7fb1a8bc6640 1 -- 172.21.15.86:0/2876171094 wait complete. 2024-03-29T11:21:50.498 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:50.520+0000 7fb1a8bc6640 1 librados: shutdown 2024-03-29T11:21:50.513 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:21:57.320 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:21:57.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.485+0000 7f88287fb640 10 monclient: get_monmap_and_config 2024-03-29T11:21:57.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.486+0000 7f88287fb640 10 monclient: build_initial_monmap 2024-03-29T11:21:57.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.486+0000 7f88287fb640 10 monclient: monmap: 2024-03-29T11:21:57.462 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:21:57.462 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:57.462 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:21:57.487001+0000 2024-03-29T11:21:57.462 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:21:57.487001+0000 2024-03-29T11:21:57.463 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:57.463 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:57.463 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:21:57.463 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:21:57.463 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:21:57.463 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:57.465 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.486+0000 7f88287fb640 1 Processor -- start 2024-03-29T11:21:57.465 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.486+0000 7f88287fb640 1 -- start start 2024-03-29T11:21:57.465 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.486+0000 7f88287fb640 10 monclient: init 2024-03-29T11:21:57.465 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.487+0000 7f88287fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:57.465 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.487+0000 7f88287fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:21:57.465 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.487+0000 7f88287fb640 10 monclient(hunting): picked mon.noname-c con 0x7f88201422b0 addr v1:172.21.15.196:6789/0 2024-03-29T11:21:57.465 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.487+0000 7f88287fb640 10 monclient(hunting): picked mon.noname-b con 0x7f8820142690 addr v1:172.21.15.86:6790/0 2024-03-29T11:21:57.465 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.487+0000 7f8826597640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f88201422b0 0x7f882013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:36632/0 (socket says 172.21.15.86:36632) 2024-03-29T11:21:57.465 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.487+0000 7f8825d96640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f8820142690 0x7f88200b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:43152/0 (socket says 172.21.15.86:43152) 2024-03-29T11:21:57.465 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.488+0000 7f88287fb640 10 monclient(hunting): picked mon.noname-a con 0x7f88200b17d0 addr v1:172.21.15.86:6789/0 2024-03-29T11:21:57.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.488+0000 7f88287fb640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8820142c80 con 0x7f88200b17d0 2024-03-29T11:21:57.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.488+0000 7f88287fb640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88200aae40 con 0x7f88201422b0 2024-03-29T11:21:57.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.488+0000 7f88287fb640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f882013fbd0 con 0x7f8820142690 2024-03-29T11:21:57.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.488+0000 7f88287fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:57.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.488+0000 7f88287fb640 10 monclient(hunting): authenticate will time out at 941.806177s 2024-03-29T11:21:57.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.488+0000 7f8826597640 1 -- 172.21.15.86:0/2691262497 learned_addr learned my addr 172.21.15.86:0/2691262497 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:21:57.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.489+0000 7f8825595640 1 -- 172.21.15.86:0/2691262497 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1279578583 0 0) 0x7f882013fbd0 con 0x7f8820142690 2024-03-29T11:21:57.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.489+0000 7f8825595640 10 monclient(hunting): _negotiate 2024-03-29T11:21:57.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.489+0000 7f8825595640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:57.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.489+0000 7f8825595640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:57.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.489+0000 7f8825595640 10 monclient(hunting): my global_id is 4283 2024-03-29T11:21:57.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.489+0000 7f8825595640 1 -- 172.21.15.86:0/2691262497 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f880c003520 con 0x7f8820142690 2024-03-29T11:21:57.470 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.489+0000 7f8825595640 1 -- 172.21.15.86:0/2691262497 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3780500384 0 0) 0x7f88200aae40 con 0x7f88201422b0 2024-03-29T11:21:57.470 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.489+0000 7f8825595640 10 monclient(hunting): _negotiate 2024-03-29T11:21:57.470 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.489+0000 7f8825595640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:57.470 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.489+0000 7f8825595640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:57.470 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.489+0000 7f8825595640 10 monclient(hunting): my global_id is 4264 2024-03-29T11:21:57.470 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.489+0000 7f8825595640 1 -- 172.21.15.86:0/2691262497 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f882013fbd0 con 0x7f88201422b0 2024-03-29T11:21:57.470 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.489+0000 7f8825595640 1 -- 172.21.15.86:0/2691262497 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 544063878 0 0) 0x7f880c003520 con 0x7f8820142690 2024-03-29T11:21:57.470 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.489+0000 7f8825595640 1 -- 172.21.15.86:0/2691262497 >> v1:172.21.15.196:6789/0 conn(0x7f88201422b0 legacy=0x7f882013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:57.470 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.490+0000 7f8825595640 1 -- 172.21.15.86:0/2691262497 >> v1:172.21.15.86:6789/0 conn(0x7f88200b17d0 legacy=0x7f88200b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:21:57.470 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.491+0000 7f8825595640 10 monclient: _finish_hunting 0 2024-03-29T11:21:57.470 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.491+0000 7f8825595640 1 monclient: found mon.noname-b 2024-03-29T11:21:57.470 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.491+0000 7f8825595640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:57.470 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.491+0000 7f8825595640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:21:57.470 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.491+0000 7f8825595640 1 -- 172.21.15.86:0/2691262497 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8820151940 con 0x7f8820142690 2024-03-29T11:21:57.471 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.491+0000 7f8825595640 10 monclient: _finish_auth 0 2024-03-29T11:21:57.471 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.491+0000 7f8825595640 10 monclient: _check_auth_tickets 2024-03-29T11:21:57.471 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.491+0000 7f8825595640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:57.471 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.491+0000 7f8825595640 1 -- 172.21.15.86:0/2691262497 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f8810001690 con 0x7f8820142690 2024-03-29T11:21:57.471 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.491+0000 7f8825595640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:57.471 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.491+0000 7f8825595640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:21:57.471 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.491+0000 7f8825595640 10 monclient: dump: 2024-03-29T11:21:57.471 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:21:57.471 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:57.471 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:57.471 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:57.471 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:57.471 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:57.471 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:21:57.472 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:21:57.472 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:21:57.472 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:57.472 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.491+0000 7f88287fb640 5 monclient: authenticate success, global_id 4283 2024-03-29T11:21:57.472 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.491+0000 7f88287fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:21:57.472 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.491+0000 7f8825595640 1 -- 172.21.15.86:0/2691262497 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88100017f0 con 0x7f8820142690 2024-03-29T11:21:57.472 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.491+0000 7f8825595640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:57.472 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.491+0000 7f8825595640 1 -- 172.21.15.86:0/2691262497 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f8810001b80 con 0x7f8820142690 2024-03-29T11:21:57.472 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.491+0000 7f8825595640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:57.472 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.491+0000 7f8825595640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:57.472 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.491+0000 7f8825595640 10 monclient: dump: 2024-03-29T11:21:57.472 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:21:57.472 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:57.472 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:57.472 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:57.473 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:57.473 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:57.473 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:21:57.473 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:21:57.473 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:21:57.473 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:57.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.491+0000 7f88287fb640 10 monclient: get_monmap_and_config success 2024-03-29T11:21:57.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.491+0000 7f88287fb640 4 set_mon_vals no callback set 2024-03-29T11:21:57.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.492+0000 7f88287fb640 10 monclient: shutdown 2024-03-29T11:21:57.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.492+0000 7f88287fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:57.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.492+0000 7f88287fb640 1 -- 172.21.15.86:0/2691262497 >> v1:172.21.15.86:6790/0 conn(0x7f8820142690 legacy=0x7f88200b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:57.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.492+0000 7f88287fb640 1 -- 172.21.15.86:0/2691262497 shutdown_connections 2024-03-29T11:21:57.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.492+0000 7f88287fb640 1 -- 172.21.15.86:0/2691262497 >> 172.21.15.86:0/2691262497 conn(0x7f8820066a00 msgr2=0x7f8820066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:57.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.493+0000 7f88287fb640 1 -- 172.21.15.86:0/2691262497 shutdown_connections 2024-03-29T11:21:57.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.493+0000 7f88287fb640 1 -- 172.21.15.86:0/2691262497 wait complete. 2024-03-29T11:21:57.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.494+0000 7f88287fb640 10 monclient: build_initial_monmap 2024-03-29T11:21:57.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.494+0000 7f88287fb640 10 monclient: monmap: 2024-03-29T11:21:57.480 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:21:57.480 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:21:57.480 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:21:57.480 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:21:57.480 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:21:57.480 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:57.480 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:21:57.480 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:21:57.480 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:21:57.480 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:57.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.494+0000 7f88287fb640 1 librados: starting msgr at 2024-03-29T11:21:57.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.494+0000 7f88287fb640 1 librados: starting objecter 2024-03-29T11:21:57.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.494+0000 7f88287fb640 1 Processor -- start 2024-03-29T11:21:57.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.494+0000 7f88287fb640 1 -- start start 2024-03-29T11:21:57.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.494+0000 7f88287fb640 1 librados: setting wanted keys 2024-03-29T11:21:57.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.494+0000 7f88287fb640 1 librados: calling monclient init 2024-03-29T11:21:57.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.494+0000 7f88287fb640 10 monclient: init 2024-03-29T11:21:57.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.495+0000 7f88287fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:21:57.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.495+0000 7f88287fb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:21:57.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.495+0000 7f88287fb640 10 monclient(hunting): picked mon.noname-b con 0x7f88200b17d0 addr v1:172.21.15.196:6789/0 2024-03-29T11:21:57.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.495+0000 7f88287fb640 10 monclient(hunting): picked mon.noname-a con 0x7f88201422b0 addr v1:172.21.15.86:6789/0 2024-03-29T11:21:57.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.495+0000 7f88287fb640 10 monclient(hunting): picked mon.noname-c con 0x7f8820141bb0 addr v1:172.21.15.86:6790/0 2024-03-29T11:21:57.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.495+0000 7f88287fb640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88200b1520 con 0x7f88201422b0 2024-03-29T11:21:57.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.495+0000 7f88287fb640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88200b6be0 con 0x7f88200b17d0 2024-03-29T11:21:57.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.495+0000 7f88287fb640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88200b6e50 con 0x7f8820141bb0 2024-03-29T11:21:57.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.495+0000 7f88287fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:21:57.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.495+0000 7f88287fb640 10 monclient(hunting): authenticate will time out at 941.813406s 2024-03-29T11:21:57.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.495+0000 7f8826d98640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f8820141bb0 0x7f88200b07c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:43742/0 (socket says 172.21.15.86:43742) 2024-03-29T11:21:57.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.495+0000 7f8816ffd640 10 client.?.objecter ms_handle_connect 0x7f88200b17d0 2024-03-29T11:21:57.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.495+0000 7f8816ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:57.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.496+0000 7f8826d98640 1 -- 172.21.15.86:0/3914894503 learned_addr learned my addr 172.21.15.86:0/3914894503 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:21:57.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.497+0000 7f8816ffd640 10 client.?.objecter ms_handle_connect 0x7f8820141bb0 2024-03-29T11:21:57.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.497+0000 7f8816ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:57.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.497+0000 7f8816ffd640 1 -- 172.21.15.86:0/3914894503 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1314333855 0 0) 0x7f88200b6be0 con 0x7f88200b17d0 2024-03-29T11:21:57.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.497+0000 7f8816ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:57.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.497+0000 7f8816ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:57.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.497+0000 7f8816ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:57.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.497+0000 7f8816ffd640 10 monclient(hunting): my global_id is 4267 2024-03-29T11:21:57.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.497+0000 7f8816ffd640 1 -- 172.21.15.86:0/3914894503 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8804004eb0 con 0x7f88200b17d0 2024-03-29T11:21:57.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.497+0000 7f8816ffd640 1 -- 172.21.15.86:0/3914894503 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1782032721 0 0) 0x7f88200b6e50 con 0x7f8820141bb0 2024-03-29T11:21:57.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.497+0000 7f8816ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:21:57.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.497+0000 7f8816ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:21:57.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.497+0000 7f8816ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:21:57.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.497+0000 7f8816ffd640 10 monclient(hunting): my global_id is 4286 2024-03-29T11:21:57.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.497+0000 7f8816ffd640 1 -- 172.21.15.86:0/3914894503 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88200b6be0 con 0x7f8820141bb0 2024-03-29T11:21:57.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.497+0000 7f8816ffd640 1 -- 172.21.15.86:0/3914894503 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 572860891 0 0) 0x7f88200b6be0 con 0x7f8820141bb0 2024-03-29T11:21:57.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.498+0000 7f8816ffd640 1 -- 172.21.15.86:0/3914894503 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88200b6e50 con 0x7f8820141bb0 2024-03-29T11:21:57.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.498+0000 7f8816ffd640 1 -- 172.21.15.86:0/3914894503 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f8818001c20 con 0x7f8820141bb0 2024-03-29T11:21:57.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.498+0000 7f8816ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:57.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.498+0000 7f8816ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:21:57.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.498+0000 7f8816ffd640 10 monclient(hunting): dump: 2024-03-29T11:21:57.484 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:21:57.484 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:57.484 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:57.484 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:57.484 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:57.484 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:57.484 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:21:57.484 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:21:57.484 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:21:57.484 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:57.484 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.498+0000 7f8816ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:21:57.484 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.498+0000 7f8816ffd640 1 -- 172.21.15.86:0/3914894503 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 421683404 0 0) 0x7f8804004eb0 con 0x7f88200b17d0 2024-03-29T11:21:57.484 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.498+0000 7f8816ffd640 1 -- 172.21.15.86:0/3914894503 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88200b6be0 con 0x7f88200b17d0 2024-03-29T11:21:57.484 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.498+0000 7f8816ffd640 1 -- 172.21.15.86:0/3914894503 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1876763474 0 0) 0x7f88200b6e50 con 0x7f8820141bb0 2024-03-29T11:21:57.484 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.498+0000 7f8816ffd640 1 -- 172.21.15.86:0/3914894503 >> v1:172.21.15.196:6789/0 conn(0x7f88200b17d0 legacy=0x7f882013fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:57.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.498+0000 7f8816ffd640 1 -- 172.21.15.86:0/3914894503 >> v1:172.21.15.86:6789/0 conn(0x7f88201422b0 legacy=0x7f88200b7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:57.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.498+0000 7f8816ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:21:57.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.498+0000 7f8816ffd640 1 monclient: found mon.c 2024-03-29T11:21:57.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.499+0000 7f8816ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:21:57.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.499+0000 7f8816ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:21:57.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.499+0000 7f8816ffd640 1 -- 172.21.15.86:0/3914894503 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88200b0ed0 con 0x7f8820141bb0 2024-03-29T11:21:57.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.499+0000 7f8816ffd640 10 monclient: _finish_auth 0 2024-03-29T11:21:57.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.499+0000 7f8816ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:21:57.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.499+0000 7f8816ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:21:57.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.499+0000 7f8816ffd640 10 client.?.objecter ms_handle_connect 0x7f88201422b0 2024-03-29T11:21:57.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.499+0000 7f8816ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:21:57.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.499+0000 7f88287fb640 5 monclient: authenticate success, global_id 4286 2024-03-29T11:21:57.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.499+0000 7f88287fb640 10 monclient: _renew_subs 2024-03-29T11:21:57.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.499+0000 7f88287fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:21:57.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.499+0000 7f88287fb640 1 -- 172.21.15.86:0/3914894503 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88200b5780 con 0x7f8820141bb0 2024-03-29T11:21:57.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.499+0000 7f88287fb640 10 client.4286.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:21:57.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.499+0000 7f88287fb640 10 monclient: _renew_subs 2024-03-29T11:21:57.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.499+0000 7f88287fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:21:57.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.499+0000 7f88287fb640 1 -- 172.21.15.86:0/3914894503 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8820150710 con 0x7f8820141bb0 2024-03-29T11:21:57.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.499+0000 7f88287fb640 1 librados: init done 2024-03-29T11:21:57.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.500+0000 7f88287fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:57.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.500+0000 7f88287fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:57.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.500+0000 7f88287fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:21:57.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.500+0000 7f88287fb640 1 -- 172.21.15.86:0/3914894503 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8820075f10 con 0x7f8820141bb0 2024-03-29T11:21:57.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.500+0000 7f8816ffd640 1 -- 172.21.15.86:0/3914894503 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8818001d80 con 0x7f8820141bb0 2024-03-29T11:21:57.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.500+0000 7f8816ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:21:57.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.500+0000 7f8824d94640 4 set_mon_vals no callback set 2024-03-29T11:21:57.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.500+0000 7f8816ffd640 1 -- 172.21.15.86:0/3914894503 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f8818003b10 con 0x7f8820141bb0 2024-03-29T11:21:57.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.500+0000 7f8816ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:21:57.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.500+0000 7f8816ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:21:57.487 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.500+0000 7f8816ffd640 10 monclient: dump: 2024-03-29T11:21:57.487 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:21:57.487 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:21:57.487 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:57.487 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:21:57.487 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:21:57.487 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:21:57.487 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:21:57.487 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:21:57.487 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:21:57.487 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:21:57.487 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.502+0000 7f8816ffd640 1 -- 172.21.15.86:0/3914894503 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f8818003c70 con 0x7f8820141bb0 2024-03-29T11:21:57.487 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.502+0000 7f8816ffd640 1 -- 172.21.15.86:0/3914894503 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f8818084bb0 con 0x7f8820141bb0 2024-03-29T11:21:57.487 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.502+0000 7f8816ffd640 10 client.4286.objecter ms_dispatch 0x7f8820000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:21:57.487 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.502+0000 7f8816ffd640 3 client.4286.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:21:57.487 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.502+0000 7f8816ffd640 3 client.4286.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:21:57.488 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.502+0000 7f8826597640 10 monclient: get_auth_request con 0x7f880406ddf0 auth_method 0 2024-03-29T11:21:57.488 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.503+0000 7f8816ffd640 20 client.4286.objecter dump_active .. 0 homeless 2024-03-29T11:21:57.488 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.504+0000 7f8816ffd640 10 client.4286.objecter ms_handle_connect 0x7f880406ddf0 2024-03-29T11:21:57.488 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.508+0000 7f8816ffd640 1 -- 172.21.15.86:0/3914894503 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f88180504f0 con 0x7f8820141bb0 2024-03-29T11:21:57.488 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.508+0000 7f8816ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:21:57.488 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.508+0000 7f8816ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:21:57.801 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.824+0000 7f88287fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:57.801 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.824+0000 7f88287fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:57.801 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.824+0000 7f88287fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:21:57.801 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.824+0000 7f88287fb640 1 -- 172.21.15.86:0/3914894503 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88200b2270 con 0x7f8820141bb0 2024-03-29T11:21:57.802 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.825+0000 7f8816ffd640 1 -- 172.21.15.86:0/3914894503 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f8818052420 con 0x7f8820141bb0 2024-03-29T11:21:57.802 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.825+0000 7f8816ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:21:57.802 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.825+0000 7f8816ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:21:57.802 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:21:57.802 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:21:57.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.828+0000 7f88287fb640 10 librados: watch_flush enter 2024-03-29T11:21:57.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.829+0000 7f88287fb640 10 librados: watch_flush exit 2024-03-29T11:21:57.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.829+0000 7f88287fb640 20 client.4286.objecter shutdown clearing up homeless session... 2024-03-29T11:21:57.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.829+0000 7f88287fb640 10 client.4286.objecter successfully canceled tick 2024-03-29T11:21:57.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.829+0000 7f88287fb640 1 -- 172.21.15.86:0/3914894503 >> v1:172.21.15.196:6816/588110318 conn(0x7f880406ddf0 legacy=0x7f88040702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:57.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.829+0000 7f88287fb640 10 monclient: shutdown 2024-03-29T11:21:57.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.829+0000 7f88287fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:21:57.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.829+0000 7f88287fb640 1 -- 172.21.15.86:0/3914894503 >> v1:172.21.15.86:6790/0 conn(0x7f8820141bb0 legacy=0x7f88200b07c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:21:57.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.829+0000 7f88287fb640 1 -- 172.21.15.86:0/3914894503 shutdown_connections 2024-03-29T11:21:57.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.829+0000 7f88287fb640 1 -- 172.21.15.86:0/3914894503 >> 172.21.15.86:0/3914894503 conn(0x7f8820066a00 msgr2=0x7f882013e9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:21:57.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.829+0000 7f88287fb640 1 -- 172.21.15.86:0/3914894503 shutdown_connections 2024-03-29T11:21:57.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.829+0000 7f88287fb640 1 -- 172.21.15.86:0/3914894503 wait complete. 2024-03-29T11:21:57.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:21:57.829+0000 7f88287fb640 1 librados: shutdown 2024-03-29T11:21:57.820 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:04.626 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:04.763 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.787+0000 7f7b4a281640 10 monclient: get_monmap_and_config 2024-03-29T11:22:04.763 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.787+0000 7f7b4a281640 10 monclient: build_initial_monmap 2024-03-29T11:22:04.763 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.787+0000 7f7b4a281640 10 monclient: monmap: 2024-03-29T11:22:04.764 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:22:04.764 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:04.764 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:22:04.788094+0000 2024-03-29T11:22:04.764 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:22:04.788094+0000 2024-03-29T11:22:04.764 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:04.764 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:04.764 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:22:04.764 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:22:04.764 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:22:04.764 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:04.765 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.787+0000 7f7b4a281640 1 Processor -- start 2024-03-29T11:22:04.765 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.788+0000 7f7b4a281640 1 -- start start 2024-03-29T11:22:04.765 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.788+0000 7f7b4a281640 10 monclient: init 2024-03-29T11:22:04.765 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.788+0000 7f7b4a281640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:04.765 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.788+0000 7f7b4a281640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:22:04.765 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.788+0000 7f7b4a281640 10 monclient(hunting): picked mon.noname-a con 0x7f7b441457f0 addr v1:172.21.15.86:6789/0 2024-03-29T11:22:04.765 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.788+0000 7f7b4a281640 10 monclient(hunting): picked mon.noname-b con 0x7f7b4413f650 addr v1:172.21.15.86:6790/0 2024-03-29T11:22:04.765 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.788+0000 7f7b4a281640 10 monclient(hunting): picked mon.noname-c con 0x7f7b440a9d70 addr v1:172.21.15.196:6789/0 2024-03-29T11:22:04.765 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.788+0000 7f7b4a281640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b440aaef0 con 0x7f7b441457f0 2024-03-29T11:22:04.766 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.788+0000 7f7b4a281640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b44145bd0 con 0x7f7b440a9d70 2024-03-29T11:22:04.766 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.788+0000 7f7b4a281640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b44145eb0 con 0x7f7b4413f650 2024-03-29T11:22:04.766 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.788+0000 7f7b4a281640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:04.766 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.788+0000 7f7b4a281640 10 monclient(hunting): authenticate will time out at 949.106908s 2024-03-29T11:22:04.766 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.789+0000 7f7b437fe640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f7b4413f650 0x7f7b440b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:47020/0 (socket says 172.21.15.86:47020) 2024-03-29T11:22:04.767 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.789+0000 7f7b437fe640 1 -- 172.21.15.86:0/181566761 learned_addr learned my addr 172.21.15.86:0/181566761 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:22:04.767 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.790+0000 7f7b42ffd640 1 -- 172.21.15.86:0/181566761 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 82121846 0 0) 0x7f7b44145eb0 con 0x7f7b4413f650 2024-03-29T11:22:04.767 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.790+0000 7f7b42ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:04.767 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.790+0000 7f7b42ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:04.767 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.790+0000 7f7b42ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:04.767 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.790+0000 7f7b42ffd640 10 monclient(hunting): my global_id is 4289 2024-03-29T11:22:04.767 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.790+0000 7f7b42ffd640 1 -- 172.21.15.86:0/181566761 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b2c003520 con 0x7f7b4413f650 2024-03-29T11:22:04.767 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.790+0000 7f7b42ffd640 1 -- 172.21.15.86:0/181566761 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3293438663 0 0) 0x7f7b2c003520 con 0x7f7b4413f650 2024-03-29T11:22:04.767 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.790+0000 7f7b42ffd640 1 -- 172.21.15.86:0/181566761 >> v1:172.21.15.196:6789/0 conn(0x7f7b440a9d70 legacy=0x7f7b4414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:04.767 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.790+0000 7f7b42ffd640 1 -- 172.21.15.86:0/181566761 >> v1:172.21.15.86:6789/0 conn(0x7f7b441457f0 legacy=0x7f7b4413ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:04.771 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.792+0000 7f7b42ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:22:04.771 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.792+0000 7f7b42ffd640 1 monclient: found mon.noname-b 2024-03-29T11:22:04.772 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.792+0000 7f7b42ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:04.772 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.792+0000 7f7b42ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:22:04.772 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.792+0000 7f7b42ffd640 1 -- 172.21.15.86:0/181566761 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b44146340 con 0x7f7b4413f650 2024-03-29T11:22:04.772 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.792+0000 7f7b42ffd640 10 monclient: _finish_auth 0 2024-03-29T11:22:04.772 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.792+0000 7f7b42ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:22:04.772 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.792+0000 7f7b42ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:04.772 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.792+0000 7f7b42ffd640 1 -- 172.21.15.86:0/181566761 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f7b38001690 con 0x7f7b4413f650 2024-03-29T11:22:04.772 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.792+0000 7f7b42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:04.772 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.792+0000 7f7b42ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:04.772 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.792+0000 7f7b42ffd640 10 monclient: dump: 2024-03-29T11:22:04.772 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:22:04.772 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:04.772 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:04.772 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:04.773 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:04.773 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:04.773 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:22:04.773 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:22:04.773 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:22:04.773 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:04.773 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.792+0000 7f7b4a281640 5 monclient: authenticate success, global_id 4289 2024-03-29T11:22:04.773 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.792+0000 7f7b4a281640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:04.773 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.792+0000 7f7b42ffd640 1 -- 172.21.15.86:0/181566761 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b38001860 con 0x7f7b4413f650 2024-03-29T11:22:04.773 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.792+0000 7f7b42ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:04.773 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.792+0000 7f7b42ffd640 1 -- 172.21.15.86:0/181566761 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f7b38001cb0 con 0x7f7b4413f650 2024-03-29T11:22:04.773 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.792+0000 7f7b42ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:04.773 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.792+0000 7f7b42ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:04.773 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.792+0000 7f7b42ffd640 10 monclient: dump: 2024-03-29T11:22:04.773 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:22:04.774 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:04.774 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:04.774 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:04.774 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:04.774 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:04.774 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:22:04.774 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:22:04.774 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:22:04.774 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:04.774 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.792+0000 7f7b4a281640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:04.774 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.792+0000 7f7b4a281640 4 set_mon_vals no callback set 2024-03-29T11:22:04.774 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.793+0000 7f7b4a281640 10 monclient: shutdown 2024-03-29T11:22:04.774 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.793+0000 7f7b4a281640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:04.774 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.793+0000 7f7b4a281640 1 -- 172.21.15.86:0/181566761 >> v1:172.21.15.86:6790/0 conn(0x7f7b4413f650 legacy=0x7f7b440b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:04.775 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.794+0000 7f7b4a281640 1 -- 172.21.15.86:0/181566761 shutdown_connections 2024-03-29T11:22:04.775 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.794+0000 7f7b4a281640 1 -- 172.21.15.86:0/181566761 >> 172.21.15.86:0/181566761 conn(0x7f7b44066a00 msgr2=0x7f7b44066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:04.775 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.794+0000 7f7b4a281640 1 -- 172.21.15.86:0/181566761 shutdown_connections 2024-03-29T11:22:04.775 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.794+0000 7f7b4a281640 1 -- 172.21.15.86:0/181566761 wait complete. 2024-03-29T11:22:04.775 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.795+0000 7f7b4a281640 10 monclient: build_initial_monmap 2024-03-29T11:22:04.775 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.795+0000 7f7b4a281640 10 monclient: monmap: 2024-03-29T11:22:04.775 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:22:04.775 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:04.775 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:22:04.775 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:22:04.775 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:04.775 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:04.775 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:22:04.775 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:22:04.775 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:22:04.776 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:04.776 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.795+0000 7f7b4a281640 1 librados: starting msgr at 2024-03-29T11:22:04.776 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.795+0000 7f7b4a281640 1 librados: starting objecter 2024-03-29T11:22:04.776 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.795+0000 7f7b4a281640 1 Processor -- start 2024-03-29T11:22:04.782 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.796+0000 7f7b4a281640 1 -- start start 2024-03-29T11:22:04.782 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.796+0000 7f7b4a281640 1 librados: setting wanted keys 2024-03-29T11:22:04.782 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.796+0000 7f7b4a281640 1 librados: calling monclient init 2024-03-29T11:22:04.782 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.796+0000 7f7b4a281640 10 monclient: init 2024-03-29T11:22:04.782 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.797+0000 7f7b4a281640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:04.782 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.797+0000 7f7b4a281640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:04.782 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.797+0000 7f7b4a281640 10 monclient(hunting): picked mon.noname-b con 0x7f7b440a9d70 addr v1:172.21.15.196:6789/0 2024-03-29T11:22:04.782 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.797+0000 7f7b4a281640 10 monclient(hunting): picked mon.noname-c con 0x7f7b4413f650 addr v1:172.21.15.86:6790/0 2024-03-29T11:22:04.782 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.797+0000 7f7b4a281640 10 monclient(hunting): picked mon.noname-a con 0x7f7b441457f0 addr v1:172.21.15.86:6789/0 2024-03-29T11:22:04.782 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.797+0000 7f7b4a281640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b441358b0 con 0x7f7b441457f0 2024-03-29T11:22:04.782 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.797+0000 7f7b4a281640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b441414a0 con 0x7f7b440a9d70 2024-03-29T11:22:04.782 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.797+0000 7f7b437fe640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f7b4413f650 0x7f7b4412f900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:47226/0 (socket says 172.21.15.86:47226) 2024-03-29T11:22:04.782 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.797+0000 7f7b437fe640 1 -- 172.21.15.86:0/253631064 learned_addr learned my addr 172.21.15.86:0/253631064 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:22:04.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.797+0000 7f7b4a281640 1 -- 172.21.15.86:0/253631064 --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b44132a30 con 0x7f7b4413f650 2024-03-29T11:22:04.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.797+0000 7f7b4a281640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:04.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.797+0000 7f7b4a281640 10 monclient(hunting): authenticate will time out at 949.115938s 2024-03-29T11:22:04.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.797+0000 7f7b40ff9640 10 client.?.objecter ms_handle_connect 0x7f7b4413f650 2024-03-29T11:22:04.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.798+0000 7f7b40ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:04.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.798+0000 7f7b40ff9640 1 -- 172.21.15.86:0/253631064 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1772221906 0 0) 0x7f7b44132a30 con 0x7f7b4413f650 2024-03-29T11:22:04.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.798+0000 7f7b40ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:22:04.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.798+0000 7f7b40ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:04.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.798+0000 7f7b40ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:04.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.798+0000 7f7b40ff9640 10 monclient(hunting): my global_id is 4292 2024-03-29T11:22:04.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.798+0000 7f7b40ff9640 1 -- 172.21.15.86:0/253631064 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b30004eb0 con 0x7f7b4413f650 2024-03-29T11:22:04.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.798+0000 7f7b40ff9640 1 -- 172.21.15.86:0/253631064 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1197287199 0 0) 0x7f7b30004eb0 con 0x7f7b4413f650 2024-03-29T11:22:04.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.798+0000 7f7b40ff9640 1 -- 172.21.15.86:0/253631064 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f7b44132a30 con 0x7f7b4413f650 2024-03-29T11:22:04.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.798+0000 7f7b40ff9640 1 -- 172.21.15.86:0/253631064 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f7b38001b70 con 0x7f7b4413f650 2024-03-29T11:22:04.784 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.798+0000 7f7b40ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:04.784 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.798+0000 7f7b40ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:04.784 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.798+0000 7f7b40ff9640 10 monclient(hunting): dump: 2024-03-29T11:22:04.784 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:22:04.784 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:04.784 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:04.784 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:04.784 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:04.784 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:04.784 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:22:04.784 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:22:04.784 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:22:04.784 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:04.784 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.798+0000 7f7b40ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:04.784 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.798+0000 7f7b40ff9640 1 -- 172.21.15.86:0/253631064 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2186597546 0 0) 0x7f7b44132a30 con 0x7f7b4413f650 2024-03-29T11:22:04.785 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.798+0000 7f7b40ff9640 1 -- 172.21.15.86:0/253631064 >> v1:172.21.15.196:6789/0 conn(0x7f7b440a9d70 legacy=0x7f7b44135db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:04.785 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.799+0000 7f7b40ff9640 1 -- 172.21.15.86:0/253631064 >> v1:172.21.15.86:6789/0 conn(0x7f7b441457f0 legacy=0x7f7b44130010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:04.785 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.799+0000 7f7b40ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:22:04.785 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.800+0000 7f7b40ff9640 1 monclient: found mon.c 2024-03-29T11:22:04.785 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.800+0000 7f7b40ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:04.785 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.800+0000 7f7b40ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:22:04.785 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.800+0000 7f7b40ff9640 1 -- 172.21.15.86:0/253631064 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b44132bc0 con 0x7f7b4413f650 2024-03-29T11:22:04.786 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.800+0000 7f7b40ff9640 10 monclient: _finish_auth 0 2024-03-29T11:22:04.786 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.800+0000 7f7b40ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:22:04.786 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.800+0000 7f7b40ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:04.786 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.800+0000 7f7b4a281640 5 monclient: authenticate success, global_id 4292 2024-03-29T11:22:04.786 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.800+0000 7f7b4a281640 10 monclient: _renew_subs 2024-03-29T11:22:04.786 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.800+0000 7f7b4a281640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:22:04.786 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.800+0000 7f7b4a281640 1 -- 172.21.15.86:0/253631064 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b441354b0 con 0x7f7b4413f650 2024-03-29T11:22:04.787 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.800+0000 7f7b40ff9640 1 -- 172.21.15.86:0/253631064 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7b38004bf0 con 0x7f7b4413f650 2024-03-29T11:22:04.787 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.801+0000 7f7b4a281640 10 client.4292.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:04.787 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.801+0000 7f7b40ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:04.787 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.801+0000 7f7b40ff9640 1 -- 172.21.15.86:0/253631064 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f7b38005040 con 0x7f7b4413f650 2024-03-29T11:22:04.787 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.801+0000 7f7b40ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:04.787 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.801+0000 7f7b40ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:04.787 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.801+0000 7f7b40ff9640 10 monclient: dump: 2024-03-29T11:22:04.787 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:22:04.787 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:04.788 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:04.788 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:04.788 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:04.788 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:04.788 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:22:04.788 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:22:04.788 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:22:04.789 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:04.789 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.801+0000 7f7b4a281640 10 monclient: _renew_subs 2024-03-29T11:22:04.789 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.801+0000 7f7b4a281640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:22:04.789 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.801+0000 7f7b4a281640 1 -- 172.21.15.86:0/253631064 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b441366a0 con 0x7f7b4413f650 2024-03-29T11:22:04.789 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.801+0000 7f7b4a281640 1 librados: init done 2024-03-29T11:22:04.789 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.801+0000 7f7b427fc640 4 set_mon_vals no callback set 2024-03-29T11:22:04.789 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.803+0000 7f7b4a281640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:04.789 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.803+0000 7f7b4a281640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:04.789 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.803+0000 7f7b4a281640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:22:04.789 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.803+0000 7f7b4a281640 1 -- 172.21.15.86:0/253631064 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b44075f10 con 0x7f7b4413f650 2024-03-29T11:22:04.789 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.803+0000 7f7b40ff9640 1 -- 172.21.15.86:0/253631064 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f7b380051a0 con 0x7f7b4413f650 2024-03-29T11:22:04.789 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.803+0000 7f7b40ff9640 1 -- 172.21.15.86:0/253631064 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f7b38084d00 con 0x7f7b4413f650 2024-03-29T11:22:04.789 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.803+0000 7f7b40ff9640 10 client.4292.objecter ms_dispatch 0x7f7b44000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:04.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.804+0000 7f7b40ff9640 3 client.4292.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:04.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.804+0000 7f7b40ff9640 3 client.4292.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:04.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.804+0000 7f7b40ff9640 20 client.4292.objecter dump_active .. 0 homeless 2024-03-29T11:22:04.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.804+0000 7f7b43fff640 10 monclient: get_auth_request con 0x7f7b3006dc90 auth_method 0 2024-03-29T11:22:04.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.804+0000 7f7b40ff9640 10 client.4292.objecter ms_handle_connect 0x7f7b3006dc90 2024-03-29T11:22:04.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.810+0000 7f7b40ff9640 1 -- 172.21.15.86:0/253631064 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f7b380505e0 con 0x7f7b4413f650 2024-03-29T11:22:04.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.810+0000 7f7b40ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:04.791 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:04.810+0000 7f7b40ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:05.107 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:05.130+0000 7f7b4a281640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:05.107 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:05.130+0000 7f7b4a281640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:05.107 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:05.130+0000 7f7b4a281640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:22:05.107 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:05.130+0000 7f7b4a281640 1 -- 172.21.15.86:0/253631064 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7b440b3680 con 0x7f7b4413f650 2024-03-29T11:22:05.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:05.133+0000 7f7b40ff9640 1 -- 172.21.15.86:0/253631064 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f7b38052510 con 0x7f7b4413f650 2024-03-29T11:22:05.110 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:05.133+0000 7f7b40ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:05.110 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:05.133+0000 7f7b40ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:05.113 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:22:05.113 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:05.114 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:05.137+0000 7f7b4a281640 10 librados: watch_flush enter 2024-03-29T11:22:05.114 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:05.137+0000 7f7b4a281640 10 librados: watch_flush exit 2024-03-29T11:22:05.114 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:05.137+0000 7f7b4a281640 20 client.4292.objecter shutdown clearing up homeless session... 2024-03-29T11:22:05.114 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:05.137+0000 7f7b4a281640 10 client.4292.objecter successfully canceled tick 2024-03-29T11:22:05.114 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:05.137+0000 7f7b4a281640 1 -- 172.21.15.86:0/253631064 >> v1:172.21.15.196:6816/588110318 conn(0x7f7b3006dc90 legacy=0x7f7b30070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:05.114 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:05.137+0000 7f7b4a281640 10 monclient: shutdown 2024-03-29T11:22:05.114 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:05.137+0000 7f7b4a281640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:05.115 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:05.137+0000 7f7b4a281640 1 -- 172.21.15.86:0/253631064 >> v1:172.21.15.86:6790/0 conn(0x7f7b4413f650 legacy=0x7f7b4412f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:05.115 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:05.137+0000 7f7b4a281640 1 -- 172.21.15.86:0/253631064 shutdown_connections 2024-03-29T11:22:05.115 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:05.137+0000 7f7b4a281640 1 -- 172.21.15.86:0/253631064 >> 172.21.15.86:0/253631064 conn(0x7f7b44066a00 msgr2=0x7f7b440b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:05.115 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:05.137+0000 7f7b4a281640 1 -- 172.21.15.86:0/253631064 shutdown_connections 2024-03-29T11:22:05.115 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:05.137+0000 7f7b4a281640 1 -- 172.21.15.86:0/253631064 wait complete. 2024-03-29T11:22:05.115 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:05.138+0000 7f7b4a281640 1 librados: shutdown 2024-03-29T11:22:05.131 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:11.937 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:12.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.111+0000 7f3b3d26d640 10 monclient: get_monmap_and_config 2024-03-29T11:22:12.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.111+0000 7f3b3d26d640 10 monclient: build_initial_monmap 2024-03-29T11:22:12.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.111+0000 7f3b3d26d640 10 monclient: monmap: 2024-03-29T11:22:12.090 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:22:12.090 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:12.090 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:22:12.112895+0000 2024-03-29T11:22:12.091 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:22:12.112895+0000 2024-03-29T11:22:12.091 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:12.091 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:12.091 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:22:12.091 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:22:12.091 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:22:12.091 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:12.093 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.112+0000 7f3b3d26d640 1 Processor -- start 2024-03-29T11:22:12.093 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.113+0000 7f3b3d26d640 1 -- start start 2024-03-29T11:22:12.093 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.113+0000 7f3b3d26d640 10 monclient: init 2024-03-29T11:22:12.093 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.113+0000 7f3b3d26d640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:12.093 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.113+0000 7f3b3d26d640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:12.093 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.113+0000 7f3b3d26d640 10 monclient(hunting): picked mon.noname-b con 0x7f3b380a9930 addr v1:172.21.15.86:6790/0 2024-03-29T11:22:12.093 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.113+0000 7f3b3d26d640 10 monclient(hunting): picked mon.noname-a con 0x7f3b38149dc0 addr v1:172.21.15.86:6789/0 2024-03-29T11:22:12.093 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.113+0000 7f3b3d26d640 10 monclient(hunting): picked mon.noname-c con 0x7f3b3814ff40 addr v1:172.21.15.196:6789/0 2024-03-29T11:22:12.093 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.113+0000 7f3b3d26d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b380aae60 con 0x7f3b38149dc0 2024-03-29T11:22:12.093 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.114+0000 7f3b3d26d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b380b4310 con 0x7f3b3814ff40 2024-03-29T11:22:12.093 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.114+0000 7f3b3d26d640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b380afdd0 con 0x7f3b380a9930 2024-03-29T11:22:12.093 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.114+0000 7f3b3d26d640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:12.093 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.114+0000 7f3b3d26d640 10 monclient(hunting): authenticate will time out at 956.432030s 2024-03-29T11:22:12.093 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.116+0000 7f3b36ffd640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f3b380a9930 0x7f3b38143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:49562/0 (socket says 172.21.15.86:49562) 2024-03-29T11:22:12.094 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.116+0000 7f3b36ffd640 1 -- 172.21.15.86:0/2088539378 learned_addr learned my addr 172.21.15.86:0/2088539378 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:22:12.096 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.118+0000 7f3b35ffb640 1 -- 172.21.15.86:0/2088539378 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2073626148 0 0) 0x7f3b380afdd0 con 0x7f3b380a9930 2024-03-29T11:22:12.096 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.118+0000 7f3b35ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:12.096 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.118+0000 7f3b35ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:12.096 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.118+0000 7f3b35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:12.096 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.118+0000 7f3b35ffb640 10 monclient(hunting): my global_id is 4295 2024-03-29T11:22:12.097 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.118+0000 7f3b35ffb640 1 -- 172.21.15.86:0/2088539378 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b20003520 con 0x7f3b380a9930 2024-03-29T11:22:12.097 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.118+0000 7f3b35ffb640 1 -- 172.21.15.86:0/2088539378 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1812030806 0 0) 0x7f3b380b4310 con 0x7f3b3814ff40 2024-03-29T11:22:12.097 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.118+0000 7f3b35ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:12.097 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.118+0000 7f3b35ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:12.097 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.118+0000 7f3b35ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:12.097 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.118+0000 7f3b35ffb640 10 monclient(hunting): my global_id is 4270 2024-03-29T11:22:12.097 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.118+0000 7f3b35ffb640 1 -- 172.21.15.86:0/2088539378 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b380afdd0 con 0x7f3b3814ff40 2024-03-29T11:22:12.097 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.120+0000 7f3b35ffb640 1 -- 172.21.15.86:0/2088539378 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1864288305 0 0) 0x7f3b20003520 con 0x7f3b380a9930 2024-03-29T11:22:12.097 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.120+0000 7f3b35ffb640 1 -- 172.21.15.86:0/2088539378 >> v1:172.21.15.196:6789/0 conn(0x7f3b3814ff40 legacy=0x7f3b38152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:12.097 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.120+0000 7f3b35ffb640 1 -- 172.21.15.86:0/2088539378 >> v1:172.21.15.86:6789/0 conn(0x7f3b38149dc0 legacy=0x7f3b3814a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:12.097 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.120+0000 7f3b35ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:12.097 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.120+0000 7f3b35ffb640 1 monclient: found mon.noname-b 2024-03-29T11:22:12.097 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.120+0000 7f3b35ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:12.097 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.120+0000 7f3b35ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:22:12.098 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.120+0000 7f3b35ffb640 1 -- 172.21.15.86:0/2088539378 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b3814a950 con 0x7f3b380a9930 2024-03-29T11:22:12.098 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.120+0000 7f3b35ffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:12.098 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.120+0000 7f3b35ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:12.098 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.120+0000 7f3b35ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:12.098 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.120+0000 7f3b3d26d640 5 monclient: authenticate success, global_id 4295 2024-03-29T11:22:12.098 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.120+0000 7f3b3d26d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:12.098 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.120+0000 7f3b35ffb640 1 -- 172.21.15.86:0/2088539378 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f3b30001940 con 0x7f3b380a9930 2024-03-29T11:22:12.098 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.120+0000 7f3b35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:12.098 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.120+0000 7f3b35ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:12.098 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.120+0000 7f3b35ffb640 10 monclient: dump: 2024-03-29T11:22:12.098 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:22:12.098 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:12.098 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:12.098 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:12.098 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:12.099 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:12.099 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:22:12.099 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:22:12.099 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:22:12.099 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:12.099 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.120+0000 7f3b3d26d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:12.101 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.120+0000 7f3b35ffb640 1 -- 172.21.15.86:0/2088539378 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b30001aa0 con 0x7f3b380a9930 2024-03-29T11:22:12.101 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.120+0000 7f3b35ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:12.101 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.120+0000 7f3b3d26d640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:12.101 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.120+0000 7f3b3d26d640 4 set_mon_vals no callback set 2024-03-29T11:22:12.101 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.121+0000 7f3b35ffb640 1 -- 172.21.15.86:0/2088539378 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f3b30001eb0 con 0x7f3b380a9930 2024-03-29T11:22:12.101 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.121+0000 7f3b35ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:12.101 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.121+0000 7f3b35ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:12.102 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.121+0000 7f3b35ffb640 10 monclient: dump: 2024-03-29T11:22:12.102 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:22:12.102 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:12.102 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:12.102 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:12.102 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:12.102 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:12.102 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:22:12.102 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:22:12.102 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:22:12.102 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:12.102 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.121+0000 7f3b3d26d640 10 monclient: shutdown 2024-03-29T11:22:12.102 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.121+0000 7f3b3d26d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:12.102 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.121+0000 7f3b3d26d640 1 -- 172.21.15.86:0/2088539378 >> v1:172.21.15.86:6790/0 conn(0x7f3b380a9930 legacy=0x7f3b38143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:12.102 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.122+0000 7f3b3d26d640 1 -- 172.21.15.86:0/2088539378 shutdown_connections 2024-03-29T11:22:12.103 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.122+0000 7f3b3d26d640 1 -- 172.21.15.86:0/2088539378 >> 172.21.15.86:0/2088539378 conn(0x7f3b380b35e0 msgr2=0x7f3b380b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:12.103 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.122+0000 7f3b3d26d640 1 -- 172.21.15.86:0/2088539378 shutdown_connections 2024-03-29T11:22:12.103 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.122+0000 7f3b3d26d640 1 -- 172.21.15.86:0/2088539378 wait complete. 2024-03-29T11:22:12.103 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.123+0000 7f3b3d26d640 10 monclient: build_initial_monmap 2024-03-29T11:22:12.103 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.123+0000 7f3b3d26d640 10 monclient: monmap: 2024-03-29T11:22:12.103 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:22:12.103 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:12.103 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:22:12.103 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:22:12.104 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:12.104 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:12.104 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:22:12.104 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:22:12.104 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:22:12.104 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:12.104 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.123+0000 7f3b3d26d640 1 librados: starting msgr at 2024-03-29T11:22:12.104 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.123+0000 7f3b3d26d640 1 librados: starting objecter 2024-03-29T11:22:12.104 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.123+0000 7f3b3d26d640 1 Processor -- start 2024-03-29T11:22:12.104 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.124+0000 7f3b3d26d640 1 -- start start 2024-03-29T11:22:12.104 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.124+0000 7f3b3d26d640 1 librados: setting wanted keys 2024-03-29T11:22:12.104 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.124+0000 7f3b3d26d640 1 librados: calling monclient init 2024-03-29T11:22:12.104 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.124+0000 7f3b3d26d640 10 monclient: init 2024-03-29T11:22:12.106 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.125+0000 7f3b3d26d640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:12.106 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.125+0000 7f3b3d26d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:12.106 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.125+0000 7f3b3d26d640 10 monclient(hunting): picked mon.noname-c con 0x7f3b380a9930 addr v1:172.21.15.86:6790/0 2024-03-29T11:22:12.106 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.125+0000 7f3b3d26d640 10 monclient(hunting): picked mon.noname-b con 0x7f3b38149dc0 addr v1:172.21.15.196:6789/0 2024-03-29T11:22:12.106 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.125+0000 7f3b3d26d640 10 monclient(hunting): picked mon.noname-a con 0x7f3b3814ff40 addr v1:172.21.15.86:6789/0 2024-03-29T11:22:12.106 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.125+0000 7f3b3d26d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b381def10 con 0x7f3b3814ff40 2024-03-29T11:22:12.106 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.125+0000 7f3b3d26d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b381dea30 con 0x7f3b38149dc0 2024-03-29T11:22:12.106 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.125+0000 7f3b3d26d640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3b381debc0 con 0x7f3b380a9930 2024-03-29T11:22:12.106 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.125+0000 7f3b3d26d640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:12.106 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.125+0000 7f3b3d26d640 10 monclient(hunting): authenticate will time out at 956.443199s 2024-03-29T11:22:12.106 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.125+0000 7f3b36ffd640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f3b380a9930 0x7f3b381df3a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:50494/0 (socket says 172.21.15.86:50494) 2024-03-29T11:22:12.107 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.125+0000 7f3b36ffd640 1 -- 172.21.15.86:0/2404094314 learned_addr learned my addr 172.21.15.86:0/2404094314 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:22:12.107 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.125+0000 7f3b1f7fe640 10 client.?.objecter ms_handle_connect 0x7f3b380a9930 2024-03-29T11:22:12.107 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.125+0000 7f3b1f7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:12.107 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.125+0000 7f3b1f7fe640 1 -- 172.21.15.86:0/2404094314 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 701784796 0 0) 0x7f3b381debc0 con 0x7f3b380a9930 2024-03-29T11:22:12.107 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.125+0000 7f3b1f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:12.107 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.125+0000 7f3b1f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:12.107 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.125+0000 7f3b1f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:12.107 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.125+0000 7f3b1f7fe640 10 monclient(hunting): my global_id is 4298 2024-03-29T11:22:12.107 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.125+0000 7f3b1f7fe640 1 -- 172.21.15.86:0/2404094314 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3b24004eb0 con 0x7f3b380a9930 2024-03-29T11:22:12.107 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.125+0000 7f3b1f7fe640 1 -- 172.21.15.86:0/2404094314 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 843295351 0 0) 0x7f3b24004eb0 con 0x7f3b380a9930 2024-03-29T11:22:12.107 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.125+0000 7f3b1f7fe640 1 -- 172.21.15.86:0/2404094314 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3b381debc0 con 0x7f3b380a9930 2024-03-29T11:22:12.107 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.125+0000 7f3b1f7fe640 1 -- 172.21.15.86:0/2404094314 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f3b30001eb0 con 0x7f3b380a9930 2024-03-29T11:22:12.107 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.125+0000 7f3b1f7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:12.107 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.125+0000 7f3b1f7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:12.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.125+0000 7f3b1f7fe640 10 monclient(hunting): dump: 2024-03-29T11:22:12.108 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:22:12.108 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:12.108 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:12.108 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:12.108 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:12.108 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:12.108 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:22:12.108 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:22:12.108 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:22:12.108 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:12.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.125+0000 7f3b1f7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:12.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.126+0000 7f3b1f7fe640 1 -- 172.21.15.86:0/2404094314 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2157549219 0 0) 0x7f3b381debc0 con 0x7f3b380a9930 2024-03-29T11:22:12.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.126+0000 7f3b1f7fe640 1 -- 172.21.15.86:0/2404094314 >> v1:172.21.15.196:6789/0 conn(0x7f3b38149dc0 legacy=0x7f3b38071b30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:12.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.126+0000 7f3b1f7fe640 1 -- 172.21.15.86:0/2404094314 >> v1:172.21.15.86:6789/0 conn(0x7f3b3814ff40 legacy=0x7f3b381e6040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:12.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.129+0000 7f3b1f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:22:12.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.129+0000 7f3b1f7fe640 1 monclient: found mon.c 2024-03-29T11:22:12.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.129+0000 7f3b1f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:12.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.129+0000 7f3b1f7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:22:12.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.129+0000 7f3b1f7fe640 1 -- 172.21.15.86:0/2404094314 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b38072240 con 0x7f3b380a9930 2024-03-29T11:22:12.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.129+0000 7f3b1f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:22:12.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.129+0000 7f3b1f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:22:12.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.129+0000 7f3b1f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:12.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.129+0000 7f3b3d26d640 5 monclient: authenticate success, global_id 4298 2024-03-29T11:22:12.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.129+0000 7f3b3d26d640 10 monclient: _renew_subs 2024-03-29T11:22:12.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.129+0000 7f3b3d26d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:22:12.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.129+0000 7f3b3d26d640 1 -- 172.21.15.86:0/2404094314 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b38072490 con 0x7f3b380a9930 2024-03-29T11:22:12.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.129+0000 7f3b3d26d640 10 client.4298.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:12.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.129+0000 7f3b3d26d640 10 monclient: _renew_subs 2024-03-29T11:22:12.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.129+0000 7f3b3d26d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:22:12.110 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.129+0000 7f3b3d26d640 1 -- 172.21.15.86:0/2404094314 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b380ab3b0 con 0x7f3b380a9930 2024-03-29T11:22:12.110 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.129+0000 7f3b3d26d640 1 librados: init done 2024-03-29T11:22:12.111 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.132+0000 7f3b3d26d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:12.111 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.132+0000 7f3b3d26d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:12.111 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.132+0000 7f3b3d26d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:22:12.111 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.132+0000 7f3b1f7fe640 1 -- 172.21.15.86:0/2404094314 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3b30004e90 con 0x7f3b380a9930 2024-03-29T11:22:12.111 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.132+0000 7f3b3d26d640 1 -- 172.21.15.86:0/2404094314 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b380715e0 con 0x7f3b380a9930 2024-03-29T11:22:12.111 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.133+0000 7f3b1f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:12.112 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.133+0000 7f3b1f7fe640 1 -- 172.21.15.86:0/2404094314 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f3b300052e0 con 0x7f3b380a9930 2024-03-29T11:22:12.112 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.133+0000 7f3b1f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:12.112 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.133+0000 7f3b1f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:12.112 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.133+0000 7f3b1f7fe640 10 monclient: dump: 2024-03-29T11:22:12.112 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:22:12.112 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:12.112 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:12.112 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:12.112 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:12.112 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:12.112 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:22:12.112 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:22:12.112 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:22:12.112 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:12.112 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.133+0000 7f3b357fa640 4 set_mon_vals no callback set 2024-03-29T11:22:12.113 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.134+0000 7f3b1f7fe640 1 -- 172.21.15.86:0/2404094314 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f3b300054c0 con 0x7f3b380a9930 2024-03-29T11:22:12.113 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.135+0000 7f3b1f7fe640 1 -- 172.21.15.86:0/2404094314 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f3b30054d60 con 0x7f3b380a9930 2024-03-29T11:22:12.114 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.135+0000 7f3b1f7fe640 10 client.4298.objecter ms_dispatch 0x7f3b38000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:12.114 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.135+0000 7f3b1f7fe640 3 client.4298.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:12.114 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.135+0000 7f3b1f7fe640 3 client.4298.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:12.114 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.135+0000 7f3b1f7fe640 20 client.4298.objecter dump_active .. 0 homeless 2024-03-29T11:22:12.114 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.136+0000 7f3b367fc640 10 monclient: get_auth_request con 0x7f3b2406db90 auth_method 0 2024-03-29T11:22:12.114 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.137+0000 7f3b1f7fe640 10 client.4298.objecter ms_handle_connect 0x7f3b2406db90 2024-03-29T11:22:12.118 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.142+0000 7f3b1f7fe640 1 -- 172.21.15.86:0/2404094314 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3b30051000 con 0x7f3b380a9930 2024-03-29T11:22:12.118 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.142+0000 7f3b1f7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:12.118 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.142+0000 7f3b1f7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:12.438 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.462+0000 7f3b3d26d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:12.438 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.462+0000 7f3b3d26d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:12.439 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.462+0000 7f3b3d26d640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:22:12.439 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.462+0000 7f3b3d26d640 1 -- 172.21.15.86:0/2404094314 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b380715e0 con 0x7f3b380a9930 2024-03-29T11:22:12.439 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.462+0000 7f3b1f7fe640 1 -- 172.21.15.86:0/2404094314 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f3b30051000 con 0x7f3b380a9930 2024-03-29T11:22:12.439 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.463+0000 7f3b1f7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:12.439 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.463+0000 7f3b1f7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:12.440 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:22:12.440 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:12.442 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.466+0000 7f3b3d26d640 10 librados: watch_flush enter 2024-03-29T11:22:12.443 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.467+0000 7f3b3d26d640 10 librados: watch_flush exit 2024-03-29T11:22:12.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.467+0000 7f3b3d26d640 20 client.4298.objecter shutdown clearing up homeless session... 2024-03-29T11:22:12.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.467+0000 7f3b3d26d640 10 client.4298.objecter successfully canceled tick 2024-03-29T11:22:12.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.467+0000 7f3b3d26d640 1 -- 172.21.15.86:0/2404094314 >> v1:172.21.15.196:6816/588110318 conn(0x7f3b2406db90 legacy=0x7f3b24070050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:12.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.467+0000 7f3b3d26d640 10 monclient: shutdown 2024-03-29T11:22:12.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.467+0000 7f3b3d26d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:12.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.467+0000 7f3b3d26d640 1 -- 172.21.15.86:0/2404094314 >> v1:172.21.15.86:6790/0 conn(0x7f3b380a9930 legacy=0x7f3b381df3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:12.445 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.468+0000 7f3b3d26d640 1 -- 172.21.15.86:0/2404094314 shutdown_connections 2024-03-29T11:22:12.445 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.468+0000 7f3b3d26d640 1 -- 172.21.15.86:0/2404094314 >> 172.21.15.86:0/2404094314 conn(0x7f3b380b35e0 msgr2=0x7f3b380b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:12.445 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.469+0000 7f3b3d26d640 1 -- 172.21.15.86:0/2404094314 shutdown_connections 2024-03-29T11:22:12.445 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.469+0000 7f3b3d26d640 1 -- 172.21.15.86:0/2404094314 wait complete. 2024-03-29T11:22:12.445 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:12.469+0000 7f3b3d26d640 1 librados: shutdown 2024-03-29T11:22:12.464 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:19.269 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:19.423 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.446+0000 7fc963fff640 10 monclient: get_monmap_and_config 2024-03-29T11:22:19.423 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.446+0000 7fc963fff640 10 monclient: build_initial_monmap 2024-03-29T11:22:19.423 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.447+0000 7fc963fff640 10 monclient: monmap: 2024-03-29T11:22:19.423 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:22:19.423 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:19.423 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:22:19.447945+0000 2024-03-29T11:22:19.423 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:22:19.447945+0000 2024-03-29T11:22:19.423 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:19.424 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:19.424 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:22:19.424 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:22:19.424 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:22:19.424 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:19.424 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.447+0000 7fc963fff640 1 Processor -- start 2024-03-29T11:22:19.424 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.447+0000 7fc963fff640 1 -- start start 2024-03-29T11:22:19.425 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.447+0000 7fc963fff640 10 monclient: init 2024-03-29T11:22:19.426 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.448+0000 7fc963fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:19.426 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.448+0000 7fc963fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:19.426 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.450+0000 7fc963fff640 10 monclient(hunting): picked mon.noname-c con 0x7fc9640ae490 addr v1:172.21.15.196:6789/0 2024-03-29T11:22:19.427 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.450+0000 7fc963fff640 10 monclient(hunting): picked mon.noname-b con 0x7fc9640aa520 addr v1:172.21.15.86:6790/0 2024-03-29T11:22:19.427 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.450+0000 7fc963fff640 10 monclient(hunting): picked mon.noname-a con 0x7fc9640ad160 addr v1:172.21.15.86:6789/0 2024-03-29T11:22:19.427 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.450+0000 7fc963fff640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9640aac70 con 0x7fc9640ad160 2024-03-29T11:22:19.427 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.450+0000 7fc963fff640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9640ac630 con 0x7fc9640ae490 2024-03-29T11:22:19.427 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.450+0000 7fc963fff640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9640aaf10 con 0x7fc9640aa520 2024-03-29T11:22:19.427 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.450+0000 7fc963fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:19.427 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.450+0000 7fc963fff640 10 monclient(hunting): authenticate will time out at 963.768841s 2024-03-29T11:22:19.427 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.451+0000 7fc9627fc640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7fc9640aa520 0x7fc9640aca50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:56982/0 (socket says 172.21.15.86:56982) 2024-03-29T11:22:19.427 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.451+0000 7fc9627fc640 1 -- 172.21.15.86:0/2119222642 learned_addr learned my addr 172.21.15.86:0/2119222642 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:22:19.430 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.454+0000 7fc961ffb640 1 -- 172.21.15.86:0/2119222642 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 657947899 0 0) 0x7fc9640aaf10 con 0x7fc9640aa520 2024-03-29T11:22:19.430 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.454+0000 7fc961ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:19.430 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.454+0000 7fc961ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:19.430 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.454+0000 7fc961ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:19.430 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.454+0000 7fc961ffb640 10 monclient(hunting): my global_id is 4301 2024-03-29T11:22:19.430 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.454+0000 7fc961ffb640 1 -- 172.21.15.86:0/2119222642 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc94c003520 con 0x7fc9640aa520 2024-03-29T11:22:19.431 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.454+0000 7fc961ffb640 1 -- 172.21.15.86:0/2119222642 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4274326625 0 0) 0x7fc9640aac70 con 0x7fc9640ad160 2024-03-29T11:22:19.431 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.454+0000 7fc961ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:19.431 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.454+0000 7fc961ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:19.432 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.454+0000 7fc961ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:19.432 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.454+0000 7fc961ffb640 10 monclient(hunting): my global_id is 4293 2024-03-29T11:22:19.432 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.454+0000 7fc961ffb640 1 -- 172.21.15.86:0/2119222642 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9640aaf10 con 0x7fc9640ad160 2024-03-29T11:22:19.432 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.454+0000 7fc961ffb640 1 -- 172.21.15.86:0/2119222642 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3107084003 0 0) 0x7fc94c003520 con 0x7fc9640aa520 2024-03-29T11:22:19.432 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.454+0000 7fc961ffb640 1 -- 172.21.15.86:0/2119222642 >> v1:172.21.15.196:6789/0 conn(0x7fc9640ae490 legacy=0x7fc964143680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:19.432 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.454+0000 7fc961ffb640 1 -- 172.21.15.86:0/2119222642 >> v1:172.21.15.86:6789/0 conn(0x7fc9640ad160 legacy=0x7fc964151ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:19.432 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.454+0000 7fc961ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:19.432 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.454+0000 7fc961ffb640 1 monclient: found mon.noname-b 2024-03-29T11:22:19.432 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.454+0000 7fc961ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:19.432 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.454+0000 7fc961ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:22:19.432 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.454+0000 7fc961ffb640 1 -- 172.21.15.86:0/2119222642 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9640aded0 con 0x7fc9640aa520 2024-03-29T11:22:19.432 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.454+0000 7fc961ffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:19.432 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.454+0000 7fc961ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:19.432 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.454+0000 7fc961ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:19.433 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.454+0000 7fc961ffb640 1 -- 172.21.15.86:0/2119222642 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc958001690 con 0x7fc9640aa520 2024-03-29T11:22:19.433 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.454+0000 7fc961ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:19.433 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.455+0000 7fc961ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:19.433 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.455+0000 7fc961ffb640 10 monclient: dump: 2024-03-29T11:22:19.433 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:22:19.433 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:19.433 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:19.433 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:19.433 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:19.433 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:19.433 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:22:19.433 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:22:19.433 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:22:19.433 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:19.433 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.455+0000 7fc963fff640 5 monclient: authenticate success, global_id 4301 2024-03-29T11:22:19.433 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.455+0000 7fc963fff640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:19.434 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.455+0000 7fc961ffb640 1 -- 172.21.15.86:0/2119222642 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc958001870 con 0x7fc9640aa520 2024-03-29T11:22:19.434 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.455+0000 7fc961ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:19.434 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.455+0000 7fc961ffb640 1 -- 172.21.15.86:0/2119222642 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc958001cc0 con 0x7fc9640aa520 2024-03-29T11:22:19.434 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.455+0000 7fc961ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:19.434 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.455+0000 7fc961ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:19.434 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.455+0000 7fc961ffb640 10 monclient: dump: 2024-03-29T11:22:19.434 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:22:19.434 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:19.434 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:19.434 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:19.434 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:19.434 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:19.434 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:22:19.434 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:22:19.434 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:22:19.435 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:19.435 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.455+0000 7fc963fff640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:19.435 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.455+0000 7fc963fff640 4 set_mon_vals no callback set 2024-03-29T11:22:19.436 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.456+0000 7fc963fff640 10 monclient: shutdown 2024-03-29T11:22:19.436 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.456+0000 7fc963fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:19.436 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.456+0000 7fc963fff640 1 -- 172.21.15.86:0/2119222642 >> v1:172.21.15.86:6790/0 conn(0x7fc9640aa520 legacy=0x7fc9640aca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:19.436 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.456+0000 7fc963fff640 1 -- 172.21.15.86:0/2119222642 shutdown_connections 2024-03-29T11:22:19.436 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.456+0000 7fc963fff640 1 -- 172.21.15.86:0/2119222642 >> 172.21.15.86:0/2119222642 conn(0x7fc9640b3a30 msgr2=0x7fc9640b3e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:19.436 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.456+0000 7fc963fff640 1 -- 172.21.15.86:0/2119222642 shutdown_connections 2024-03-29T11:22:19.436 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.456+0000 7fc963fff640 1 -- 172.21.15.86:0/2119222642 wait complete. 2024-03-29T11:22:19.436 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.457+0000 7fc963fff640 10 monclient: build_initial_monmap 2024-03-29T11:22:19.437 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.457+0000 7fc963fff640 10 monclient: monmap: 2024-03-29T11:22:19.437 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:22:19.437 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:19.437 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:22:19.437 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:22:19.437 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:19.437 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:19.437 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:22:19.438 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:22:19.438 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:22:19.438 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:19.438 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.457+0000 7fc963fff640 1 librados: starting msgr at 2024-03-29T11:22:19.438 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.457+0000 7fc963fff640 1 librados: starting objecter 2024-03-29T11:22:19.438 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.458+0000 7fc963fff640 1 Processor -- start 2024-03-29T11:22:19.438 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.458+0000 7fc963fff640 1 -- start start 2024-03-29T11:22:19.439 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.458+0000 7fc963fff640 1 librados: setting wanted keys 2024-03-29T11:22:19.439 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.458+0000 7fc963fff640 1 librados: calling monclient init 2024-03-29T11:22:19.439 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.458+0000 7fc963fff640 10 monclient: init 2024-03-29T11:22:19.439 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.459+0000 7fc963fff640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:19.439 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.459+0000 7fc963fff640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:19.440 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.460+0000 7fc963fff640 10 monclient(hunting): picked mon.noname-b con 0x7fc9640aa520 addr v1:172.21.15.196:6789/0 2024-03-29T11:22:19.440 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.460+0000 7fc963fff640 10 monclient(hunting): picked mon.noname-c con 0x7fc9640ad160 addr v1:172.21.15.86:6790/0 2024-03-29T11:22:19.440 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.460+0000 7fc963fff640 10 monclient(hunting): picked mon.noname-a con 0x7fc9640ae490 addr v1:172.21.15.86:6789/0 2024-03-29T11:22:19.441 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.460+0000 7fc963fff640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc964072310 con 0x7fc9640ae490 2024-03-29T11:22:19.441 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.460+0000 7fc963fff640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9640724a0 con 0x7fc9640aa520 2024-03-29T11:22:19.441 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.460+0000 7fc963fff640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc96413e8a0 con 0x7fc9640ad160 2024-03-29T11:22:19.441 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.460+0000 7fc963fff640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:19.441 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.460+0000 7fc963fff640 10 monclient(hunting): authenticate will time out at 963.778683s 2024-03-29T11:22:19.441 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.462+0000 7fc9627fc640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7fc9640ad160 0x7fc9641de880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:43238/0 (socket says 172.21.15.86:43238) 2024-03-29T11:22:19.441 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.462+0000 7fc9627fc640 1 -- 172.21.15.86:0/4176004418 learned_addr learned my addr 172.21.15.86:0/4176004418 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:22:19.442 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.462+0000 7fc94b7fe640 10 client.?.objecter ms_handle_connect 0x7fc9640ad160 2024-03-29T11:22:19.442 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.462+0000 7fc94b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:19.442 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.465+0000 7fc94b7fe640 10 client.?.objecter ms_handle_connect 0x7fc9640aa520 2024-03-29T11:22:19.443 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.465+0000 7fc94b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:19.443 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.466+0000 7fc94b7fe640 10 client.?.objecter ms_handle_connect 0x7fc9640ae490 2024-03-29T11:22:19.443 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.466+0000 7fc94b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:19.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.468+0000 7fc94b7fe640 1 -- 172.21.15.86:0/4176004418 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1309588250 0 0) 0x7fc964072310 con 0x7fc9640ae490 2024-03-29T11:22:19.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.468+0000 7fc94b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:19.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.468+0000 7fc94b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:19.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.468+0000 7fc94b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:19.445 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.468+0000 7fc94b7fe640 10 monclient(hunting): my global_id is 4296 2024-03-29T11:22:19.445 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.468+0000 7fc94b7fe640 1 -- 172.21.15.86:0/4176004418 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc950004eb0 con 0x7fc9640ae490 2024-03-29T11:22:19.450 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.468+0000 7fc94b7fe640 1 -- 172.21.15.86:0/4176004418 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2221944964 0 0) 0x7fc950004eb0 con 0x7fc9640ae490 2024-03-29T11:22:19.450 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.468+0000 7fc94b7fe640 1 -- 172.21.15.86:0/4176004418 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc964072310 con 0x7fc9640ae490 2024-03-29T11:22:19.450 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.468+0000 7fc94b7fe640 1 -- 172.21.15.86:0/4176004418 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc95c001730 con 0x7fc9640ae490 2024-03-29T11:22:19.450 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.468+0000 7fc94b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:19.450 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.468+0000 7fc94b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:19.450 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.468+0000 7fc94b7fe640 10 monclient(hunting): dump: 2024-03-29T11:22:19.450 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:22:19.453 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:19.454 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:19.454 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:19.454 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:19.454 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:19.454 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:22:19.454 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:22:19.454 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:22:19.454 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:19.454 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.468+0000 7fc94b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:19.454 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.468+0000 7fc94b7fe640 1 -- 172.21.15.86:0/4176004418 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3376666911 0 0) 0x7fc9640724a0 con 0x7fc9640aa520 2024-03-29T11:22:19.454 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.468+0000 7fc94b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:19.454 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.468+0000 7fc94b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:19.455 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.468+0000 7fc94b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:19.455 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.468+0000 7fc94b7fe640 10 monclient(hunting): my global_id is 4276 2024-03-29T11:22:19.455 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.468+0000 7fc94b7fe640 1 -- 172.21.15.86:0/4176004418 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc950004eb0 con 0x7fc9640aa520 2024-03-29T11:22:19.455 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.469+0000 7fc94b7fe640 1 -- 172.21.15.86:0/4176004418 <== mon.0 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2729927753 0 0) 0x7fc964072310 con 0x7fc9640ae490 2024-03-29T11:22:19.455 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.469+0000 7fc94b7fe640 1 -- 172.21.15.86:0/4176004418 >> v1:172.21.15.86:6790/0 conn(0x7fc9640ad160 legacy=0x7fc9641de880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:19.455 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.469+0000 7fc94b7fe640 1 -- 172.21.15.86:0/4176004418 >> v1:172.21.15.196:6789/0 conn(0x7fc9640aa520 legacy=0x7fc96413e190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:19.455 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.469+0000 7fc94b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:22:19.455 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.469+0000 7fc94b7fe640 1 monclient: found mon.a 2024-03-29T11:22:19.455 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.469+0000 7fc94b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:19.455 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.469+0000 7fc94b7fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:22:19.455 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.469+0000 7fc94b7fe640 1 -- 172.21.15.86:0/4176004418 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc96413ea30 con 0x7fc9640ae490 2024-03-29T11:22:19.455 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.469+0000 7fc94b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:22:19.455 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.469+0000 7fc94b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:22:19.455 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.469+0000 7fc94b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:19.455 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.469+0000 7fc963fff640 5 monclient: authenticate success, global_id 4296 2024-03-29T11:22:19.455 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.469+0000 7fc963fff640 10 monclient: _renew_subs 2024-03-29T11:22:19.456 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.469+0000 7fc963fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:22:19.456 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.469+0000 7fc963fff640 1 -- 172.21.15.86:0/4176004418 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9641def90 con 0x7fc9640ae490 2024-03-29T11:22:19.456 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.469+0000 7fc963fff640 10 client.4296.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:19.456 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.469+0000 7fc963fff640 10 monclient: _renew_subs 2024-03-29T11:22:19.456 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.469+0000 7fc963fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:22:19.456 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.469+0000 7fc963fff640 1 -- 172.21.15.86:0/4176004418 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9641459e0 con 0x7fc9640ae490 2024-03-29T11:22:19.456 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.470+0000 7fc963fff640 1 librados: init done 2024-03-29T11:22:19.456 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.471+0000 7fc94b7fe640 1 -- 172.21.15.86:0/4176004418 <== mon.0 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc95c003970 con 0x7fc9640ae490 2024-03-29T11:22:19.456 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.471+0000 7fc94b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:19.456 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.471+0000 7fc94b7fe640 1 -- 172.21.15.86:0/4176004418 <== mon.0 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc95c003c80 con 0x7fc9640ae490 2024-03-29T11:22:19.456 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.471+0000 7fc94b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:19.456 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.471+0000 7fc94b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:22:19.456 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.471+0000 7fc94b7fe640 10 monclient: dump: 2024-03-29T11:22:19.456 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:22:19.456 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:19.457 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:19.457 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:19.457 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:19.457 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:19.457 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:22:19.457 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:22:19.457 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:22:19.457 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:19.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.472+0000 7fc94b7fe640 1 -- 172.21.15.86:0/4176004418 <== mon.0 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7fc95c019cd0 con 0x7fc9640ae490 2024-03-29T11:22:19.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.472+0000 7fc9617fa640 4 set_mon_vals no callback set 2024-03-29T11:22:19.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.472+0000 7fc94b7fe640 1 -- 172.21.15.86:0/4176004418 <== mon.0 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7fc95c083ba0 con 0x7fc9640ae490 2024-03-29T11:22:19.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.472+0000 7fc962ffd640 10 monclient: get_auth_request con 0x7fc95006de70 auth_method 0 2024-03-29T11:22:19.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.472+0000 7fc94b7fe640 10 client.4296.objecter ms_dispatch 0x7fc964000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:19.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.472+0000 7fc94b7fe640 3 client.4296.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:19.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.472+0000 7fc94b7fe640 3 client.4296.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:19.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.473+0000 7fc94b7fe640 20 client.4296.objecter dump_active .. 0 homeless 2024-03-29T11:22:19.458 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.473+0000 7fc94b7fe640 10 client.4296.objecter ms_handle_connect 0x7fc95006de70 2024-03-29T11:22:19.458 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.474+0000 7fc963fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:19.458 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.474+0000 7fc963fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:19.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.474+0000 7fc963fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:22:19.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.474+0000 7fc963fff640 1 -- 172.21.15.86:0/4176004418 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc944005080 con 0x7fc9640ae490 2024-03-29T11:22:19.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.482+0000 7fc94b7fe640 1 -- 172.21.15.86:0/4176004418 <== mon.0 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc95c04f480 con 0x7fc9640ae490 2024-03-29T11:22:19.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.482+0000 7fc94b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:19.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.482+0000 7fc94b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:19.799 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.822+0000 7fc963fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:19.799 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.822+0000 7fc963fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:19.799 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.822+0000 7fc963fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:22:19.799 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.822+0000 7fc963fff640 1 -- 172.21.15.86:0/4176004418 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc944005330 con 0x7fc9640ae490 2024-03-29T11:22:19.801 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.825+0000 7fc94b7fe640 1 -- 172.21.15.86:0/4176004418 <== mon.0 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7fc95c0513b0 con 0x7fc9640ae490 2024-03-29T11:22:19.801 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.825+0000 7fc94b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:19.801 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.825+0000 7fc94b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:19.801 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:22:19.801 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:19.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.828+0000 7fc963fff640 10 librados: watch_flush enter 2024-03-29T11:22:19.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.829+0000 7fc963fff640 10 librados: watch_flush exit 2024-03-29T11:22:19.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.829+0000 7fc963fff640 20 client.4296.objecter shutdown clearing up homeless session... 2024-03-29T11:22:19.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.829+0000 7fc963fff640 10 client.4296.objecter successfully canceled tick 2024-03-29T11:22:19.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.829+0000 7fc963fff640 1 -- 172.21.15.86:0/4176004418 >> v1:172.21.15.196:6816/588110318 conn(0x7fc95006de70 legacy=0x7fc950070330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:19.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.829+0000 7fc963fff640 10 monclient: shutdown 2024-03-29T11:22:19.807 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.829+0000 7fc963fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:19.807 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.829+0000 7fc963fff640 1 -- 172.21.15.86:0/4176004418 >> v1:172.21.15.86:6789/0 conn(0x7fc9640ae490 legacy=0x7fc9641452d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:19.807 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.829+0000 7fc963fff640 1 -- 172.21.15.86:0/4176004418 shutdown_connections 2024-03-29T11:22:19.807 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.830+0000 7fc963fff640 1 -- 172.21.15.86:0/4176004418 >> 172.21.15.86:0/4176004418 conn(0x7fc9640b3a30 msgr2=0x7fc9640b3e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:19.807 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.830+0000 7fc963fff640 1 -- 172.21.15.86:0/4176004418 shutdown_connections 2024-03-29T11:22:19.807 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.830+0000 7fc963fff640 1 -- 172.21.15.86:0/4176004418 wait complete. 2024-03-29T11:22:19.807 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:19.831+0000 7fc963fff640 1 librados: shutdown 2024-03-29T11:22:19.826 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:26.634 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:26.791 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.814+0000 7f2b538f6640 10 monclient: get_monmap_and_config 2024-03-29T11:22:26.792 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.814+0000 7f2b538f6640 10 monclient: build_initial_monmap 2024-03-29T11:22:26.792 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.814+0000 7f2b538f6640 10 monclient: monmap: 2024-03-29T11:22:26.792 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:22:26.792 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:26.792 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:22:26.815437+0000 2024-03-29T11:22:26.792 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:22:26.815437+0000 2024-03-29T11:22:26.792 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:26.792 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:26.792 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:22:26.792 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:22:26.792 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:22:26.792 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:26.795 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.816+0000 7f2b538f6640 1 Processor -- start 2024-03-29T11:22:26.795 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.817+0000 7f2b538f6640 1 -- start start 2024-03-29T11:22:26.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.817+0000 7f2b538f6640 10 monclient: init 2024-03-29T11:22:26.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.817+0000 7f2b538f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:26.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.817+0000 7f2b538f6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:22:26.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.817+0000 7f2b538f6640 10 monclient(hunting): picked mon.noname-c con 0x7f2b4c0a9930 addr v1:172.21.15.196:6789/0 2024-03-29T11:22:26.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.817+0000 7f2b538f6640 10 monclient(hunting): picked mon.noname-b con 0x7f2b4c149dc0 addr v1:172.21.15.86:6790/0 2024-03-29T11:22:26.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.817+0000 7f2b538f6640 10 monclient(hunting): picked mon.noname-a con 0x7f2b4c14ff40 addr v1:172.21.15.86:6789/0 2024-03-29T11:22:26.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.817+0000 7f2b538f6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b4c0aae60 con 0x7f2b4c14ff40 2024-03-29T11:22:26.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.817+0000 7f2b538f6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b4c0b4310 con 0x7f2b4c0a9930 2024-03-29T11:22:26.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.817+0000 7f2b538f6640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b4c0afdd0 con 0x7f2b4c149dc0 2024-03-29T11:22:26.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.817+0000 7f2b538f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:26.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.817+0000 7f2b538f6640 10 monclient(hunting): authenticate will time out at 971.135996s 2024-03-29T11:22:26.799 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.822+0000 7f2b51692640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f2b4c0a9930 0x7f2b4c143a30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:56299/0 (socket says 172.21.15.86:56299) 2024-03-29T11:22:26.800 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.822+0000 7f2b51692640 1 -- 172.21.15.86:0/3728040566 learned_addr learned my addr 172.21.15.86:0/3728040566 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:22:26.800 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.823+0000 7f2b43fff640 1 -- 172.21.15.86:0/3728040566 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2338263771 0 0) 0x7f2b4c0b4310 con 0x7f2b4c0a9930 2024-03-29T11:22:26.800 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.823+0000 7f2b43fff640 10 monclient(hunting): _negotiate 2024-03-29T11:22:26.801 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.823+0000 7f2b43fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:26.801 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.823+0000 7f2b43fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:26.801 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.823+0000 7f2b43fff640 10 monclient(hunting): my global_id is 4279 2024-03-29T11:22:26.801 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.823+0000 7f2b43fff640 1 -- 172.21.15.86:0/3728040566 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b38003520 con 0x7f2b4c0a9930 2024-03-29T11:22:26.801 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.824+0000 7f2b43fff640 1 -- 172.21.15.86:0/3728040566 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 540320868 0 0) 0x7f2b38003520 con 0x7f2b4c0a9930 2024-03-29T11:22:26.801 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.824+0000 7f2b43fff640 1 -- 172.21.15.86:0/3728040566 >> v1:172.21.15.86:6790/0 conn(0x7f2b4c149dc0 legacy=0x7f2b4c14a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:26.801 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.824+0000 7f2b43fff640 1 -- 172.21.15.86:0/3728040566 >> v1:172.21.15.86:6789/0 conn(0x7f2b4c14ff40 legacy=0x7f2b4c152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:26.801 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.824+0000 7f2b43fff640 10 monclient: _finish_hunting 0 2024-03-29T11:22:26.801 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.824+0000 7f2b43fff640 1 monclient: found mon.noname-c 2024-03-29T11:22:26.802 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.824+0000 7f2b43fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:26.802 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.824+0000 7f2b43fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T11:22:26.802 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.824+0000 7f2b43fff640 1 -- 172.21.15.86:0/3728040566 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b4c14a950 con 0x7f2b4c0a9930 2024-03-29T11:22:26.802 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.824+0000 7f2b43fff640 10 monclient: _finish_auth 0 2024-03-29T11:22:26.802 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.824+0000 7f2b43fff640 10 monclient: _check_auth_tickets 2024-03-29T11:22:26.802 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.824+0000 7f2b43fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:26.802 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.824+0000 7f2b43fff640 1 -- 172.21.15.86:0/3728040566 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f2b48001670 con 0x7f2b4c0a9930 2024-03-29T11:22:26.802 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.824+0000 7f2b43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:26.802 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.824+0000 7f2b43fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:26.802 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.824+0000 7f2b43fff640 10 monclient: dump: 2024-03-29T11:22:26.802 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:22:26.802 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:26.802 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:26.803 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:26.803 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:26.803 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:26.803 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:22:26.803 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:22:26.803 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:22:26.803 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:26.803 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.824+0000 7f2b43fff640 1 -- 172.21.15.86:0/3728040566 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b480018d0 con 0x7f2b4c0a9930 2024-03-29T11:22:26.803 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.824+0000 7f2b43fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:26.803 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.824+0000 7f2b43fff640 1 -- 172.21.15.86:0/3728040566 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f2b48001670 con 0x7f2b4c0a9930 2024-03-29T11:22:26.803 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.824+0000 7f2b43fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:26.803 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.824+0000 7f2b43fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:26.803 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.824+0000 7f2b43fff640 10 monclient: dump: 2024-03-29T11:22:26.803 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:22:26.804 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:26.804 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:26.804 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:26.804 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:26.804 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:26.804 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:22:26.804 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:22:26.804 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:22:26.804 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:26.804 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.824+0000 7f2b538f6640 5 monclient: authenticate success, global_id 4279 2024-03-29T11:22:26.804 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.824+0000 7f2b538f6640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:26.804 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.824+0000 7f2b538f6640 4 set_mon_vals no callback set 2024-03-29T11:22:26.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.825+0000 7f2b538f6640 10 monclient: shutdown 2024-03-29T11:22:26.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.825+0000 7f2b538f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:26.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.825+0000 7f2b538f6640 1 -- 172.21.15.86:0/3728040566 >> v1:172.21.15.196:6789/0 conn(0x7f2b4c0a9930 legacy=0x7f2b4c143a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:26.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.828+0000 7f2b538f6640 1 -- 172.21.15.86:0/3728040566 shutdown_connections 2024-03-29T11:22:26.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.828+0000 7f2b538f6640 1 -- 172.21.15.86:0/3728040566 >> 172.21.15.86:0/3728040566 conn(0x7f2b4c0b35e0 msgr2=0x7f2b4c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:26.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.828+0000 7f2b538f6640 1 -- 172.21.15.86:0/3728040566 shutdown_connections 2024-03-29T11:22:26.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.829+0000 7f2b538f6640 1 -- 172.21.15.86:0/3728040566 wait complete. 2024-03-29T11:22:26.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.830+0000 7f2b538f6640 10 monclient: build_initial_monmap 2024-03-29T11:22:26.809 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.830+0000 7f2b538f6640 10 monclient: monmap: 2024-03-29T11:22:26.809 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:22:26.809 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:26.810 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:22:26.810 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:22:26.810 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:26.810 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:26.810 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:22:26.810 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:22:26.810 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:22:26.810 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:26.810 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.830+0000 7f2b538f6640 1 librados: starting msgr at 2024-03-29T11:22:26.810 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.830+0000 7f2b538f6640 1 librados: starting objecter 2024-03-29T11:22:26.810 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.830+0000 7f2b538f6640 1 Processor -- start 2024-03-29T11:22:26.810 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.830+0000 7f2b538f6640 1 -- start start 2024-03-29T11:22:26.810 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.830+0000 7f2b538f6640 1 librados: setting wanted keys 2024-03-29T11:22:26.810 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.830+0000 7f2b538f6640 1 librados: calling monclient init 2024-03-29T11:22:26.811 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.830+0000 7f2b538f6640 10 monclient: init 2024-03-29T11:22:26.811 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.831+0000 7f2b538f6640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:26.811 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.831+0000 7f2b538f6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:26.811 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.831+0000 7f2b538f6640 10 monclient(hunting): picked mon.noname-b con 0x7f2b4c0a9930 addr v1:172.21.15.196:6789/0 2024-03-29T11:22:26.811 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.831+0000 7f2b538f6640 10 monclient(hunting): picked mon.noname-a con 0x7f2b4c149dc0 addr v1:172.21.15.86:6789/0 2024-03-29T11:22:26.811 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.831+0000 7f2b538f6640 10 monclient(hunting): picked mon.noname-c con 0x7f2b4c14ff40 addr v1:172.21.15.86:6790/0 2024-03-29T11:22:26.811 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.831+0000 7f2b538f6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b4c1e8290 con 0x7f2b4c149dc0 2024-03-29T11:22:26.811 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.831+0000 7f2b538f6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b4c072510 con 0x7f2b4c0a9930 2024-03-29T11:22:26.811 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.831+0000 7f2b538f6640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b4c1e7ea0 con 0x7f2b4c14ff40 2024-03-29T11:22:26.811 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.831+0000 7f2b538f6640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:26.811 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.831+0000 7f2b538f6640 10 monclient(hunting): authenticate will time out at 971.149642s 2024-03-29T11:22:26.814 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.835+0000 7f2b51e93640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f2b4c14ff40 0x7f2b4c1450a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:47257/0 (socket says 172.21.15.86:47257) 2024-03-29T11:22:26.814 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.835+0000 7f2b51e93640 1 -- 172.21.15.86:0/4211902992 learned_addr learned my addr 172.21.15.86:0/4211902992 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:22:26.814 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.836+0000 7f2b41ffb640 10 client.?.objecter ms_handle_connect 0x7f2b4c14ff40 2024-03-29T11:22:26.814 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.836+0000 7f2b41ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:26.814 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.836+0000 7f2b41ffb640 1 -- 172.21.15.86:0/4211902992 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 189430677 0 0) 0x7f2b4c1e7ea0 con 0x7f2b4c14ff40 2024-03-29T11:22:26.814 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.836+0000 7f2b41ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:26.814 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.836+0000 7f2b41ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:26.814 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.836+0000 7f2b41ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:26.814 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.836+0000 7f2b41ffb640 10 monclient(hunting): my global_id is 4307 2024-03-29T11:22:26.814 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.836+0000 7f2b41ffb640 1 -- 172.21.15.86:0/4211902992 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b30004eb0 con 0x7f2b4c14ff40 2024-03-29T11:22:26.815 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.836+0000 7f2b41ffb640 10 client.?.objecter ms_handle_connect 0x7f2b4c0a9930 2024-03-29T11:22:26.815 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.836+0000 7f2b41ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:26.815 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.836+0000 7f2b41ffb640 1 -- 172.21.15.86:0/4211902992 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2880132212 0 0) 0x7f2b30004eb0 con 0x7f2b4c14ff40 2024-03-29T11:22:26.815 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.837+0000 7f2b41ffb640 1 -- 172.21.15.86:0/4211902992 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2b4c1e7ea0 con 0x7f2b4c14ff40 2024-03-29T11:22:26.815 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.837+0000 7f2b41ffb640 1 -- 172.21.15.86:0/4211902992 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f2b44001ad0 con 0x7f2b4c14ff40 2024-03-29T11:22:26.815 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.837+0000 7f2b41ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:26.815 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.837+0000 7f2b41ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:26.815 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.837+0000 7f2b41ffb640 10 monclient(hunting): dump: 2024-03-29T11:22:26.815 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:22:26.815 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:26.815 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:26.815 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:26.815 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:26.815 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:26.815 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:22:26.816 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:22:26.816 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:22:26.816 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:26.816 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.837+0000 7f2b41ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:26.816 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.837+0000 7f2b41ffb640 1 -- 172.21.15.86:0/4211902992 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4039567269 0 0) 0x7f2b4c1e7ea0 con 0x7f2b4c14ff40 2024-03-29T11:22:26.816 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.837+0000 7f2b41ffb640 1 -- 172.21.15.86:0/4211902992 >> v1:172.21.15.196:6789/0 conn(0x7f2b4c0a9930 legacy=0x7f2b4c1e7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:26.816 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.837+0000 7f2b41ffb640 1 -- 172.21.15.86:0/4211902992 >> v1:172.21.15.86:6789/0 conn(0x7f2b4c149dc0 legacy=0x7f2b4c070d60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:26.819 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.838+0000 7f2b41ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:26.819 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.838+0000 7f2b41ffb640 1 monclient: found mon.c 2024-03-29T11:22:26.819 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.838+0000 7f2b41ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:26.819 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.838+0000 7f2b41ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:22:26.819 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.838+0000 7f2b41ffb640 1 -- 172.21.15.86:0/4211902992 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b4c1e8030 con 0x7f2b4c14ff40 2024-03-29T11:22:26.819 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.838+0000 7f2b41ffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:26.819 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.838+0000 7f2b41ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:26.819 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.838+0000 7f2b41ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:26.820 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.838+0000 7f2b538f6640 5 monclient: authenticate success, global_id 4307 2024-03-29T11:22:26.820 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.838+0000 7f2b538f6640 10 monclient: _renew_subs 2024-03-29T11:22:26.820 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.838+0000 7f2b538f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:22:26.820 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.838+0000 7f2b538f6640 1 -- 172.21.15.86:0/4211902992 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b4c071a70 con 0x7f2b4c14ff40 2024-03-29T11:22:26.820 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.839+0000 7f2b538f6640 10 client.4307.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:26.820 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.839+0000 7f2b538f6640 10 monclient: _renew_subs 2024-03-29T11:22:26.820 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.839+0000 7f2b538f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:22:26.820 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.839+0000 7f2b538f6640 1 -- 172.21.15.86:0/4211902992 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b4c071470 con 0x7f2b4c14ff40 2024-03-29T11:22:26.820 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.839+0000 7f2b538f6640 1 librados: init done 2024-03-29T11:22:26.820 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.840+0000 7f2b538f6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:26.820 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.840+0000 7f2b538f6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:26.820 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.840+0000 7f2b538f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:22:26.821 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.840+0000 7f2b538f6640 1 -- 172.21.15.86:0/4211902992 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b4c1e84d0 con 0x7f2b4c14ff40 2024-03-29T11:22:26.821 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.840+0000 7f2b41ffb640 1 -- 172.21.15.86:0/4211902992 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2b44002060 con 0x7f2b4c14ff40 2024-03-29T11:22:26.821 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.840+0000 7f2b41ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:26.821 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.840+0000 7f2b41ffb640 1 -- 172.21.15.86:0/4211902992 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f2b44004d20 con 0x7f2b4c14ff40 2024-03-29T11:22:26.821 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.840+0000 7f2b41ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:26.821 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.840+0000 7f2b41ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:26.821 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.840+0000 7f2b41ffb640 10 monclient: dump: 2024-03-29T11:22:26.821 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:22:26.821 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:26.821 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:26.821 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:26.821 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:26.821 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:26.821 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:22:26.822 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:22:26.822 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:22:26.822 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:26.822 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.841+0000 7f2b437fe640 4 set_mon_vals no callback set 2024-03-29T11:22:26.823 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.843+0000 7f2b41ffb640 1 -- 172.21.15.86:0/4211902992 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f2b44004e80 con 0x7f2b4c14ff40 2024-03-29T11:22:26.823 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.843+0000 7f2b41ffb640 1 -- 172.21.15.86:0/4211902992 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f2b44084ad0 con 0x7f2b4c14ff40 2024-03-29T11:22:26.823 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.843+0000 7f2b41ffb640 10 client.4307.objecter ms_dispatch 0x7f2b4c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:26.823 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.843+0000 7f2b41ffb640 3 client.4307.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:26.823 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.843+0000 7f2b41ffb640 3 client.4307.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:26.823 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.843+0000 7f2b41ffb640 20 client.4307.objecter dump_active .. 0 homeless 2024-03-29T11:22:26.824 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.846+0000 7f2b51692640 10 monclient: get_auth_request con 0x7f2b3006dd40 auth_method 0 2024-03-29T11:22:26.824 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.848+0000 7f2b41ffb640 10 client.4307.objecter ms_handle_connect 0x7f2b3006dd40 2024-03-29T11:22:26.825 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.848+0000 7f2b41ffb640 1 -- 172.21.15.86:0/4211902992 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2b440503b0 con 0x7f2b4c14ff40 2024-03-29T11:22:26.825 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.849+0000 7f2b41ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:26.825 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:26.849+0000 7f2b41ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:27.154 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:27.177+0000 7f2b538f6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:27.154 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:27.177+0000 7f2b538f6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:27.154 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:27.177+0000 7f2b538f6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:22:27.154 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:27.177+0000 7f2b538f6640 1 -- 172.21.15.86:0/4211902992 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b4c1e84d0 con 0x7f2b4c14ff40 2024-03-29T11:22:27.155 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:27.178+0000 7f2b41ffb640 1 -- 172.21.15.86:0/4211902992 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f2b4c1e84d0 con 0x7f2b4c14ff40 2024-03-29T11:22:27.155 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:27.178+0000 7f2b41ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:27.156 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:27.178+0000 7f2b41ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:27.156 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:22:27.156 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:27.160 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:27.182+0000 7f2b538f6640 10 librados: watch_flush enter 2024-03-29T11:22:27.160 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:27.182+0000 7f2b538f6640 10 librados: watch_flush exit 2024-03-29T11:22:27.160 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:27.182+0000 7f2b538f6640 20 client.4307.objecter shutdown clearing up homeless session... 2024-03-29T11:22:27.160 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:27.182+0000 7f2b538f6640 10 client.4307.objecter successfully canceled tick 2024-03-29T11:22:27.164 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:27.185+0000 7f2b538f6640 1 -- 172.21.15.86:0/4211902992 >> v1:172.21.15.196:6816/588110318 conn(0x7f2b3006dd40 legacy=0x7f2b30070200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:27.164 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:27.185+0000 7f2b538f6640 10 monclient: shutdown 2024-03-29T11:22:27.164 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:27.185+0000 7f2b538f6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:27.164 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:27.185+0000 7f2b538f6640 1 -- 172.21.15.86:0/4211902992 >> v1:172.21.15.86:6790/0 conn(0x7f2b4c14ff40 legacy=0x7f2b4c1450a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:27.164 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:27.186+0000 7f2b538f6640 1 -- 172.21.15.86:0/4211902992 shutdown_connections 2024-03-29T11:22:27.164 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:27.186+0000 7f2b538f6640 1 -- 172.21.15.86:0/4211902992 >> 172.21.15.86:0/4211902992 conn(0x7f2b4c0b35e0 msgr2=0x7f2b4c0b2780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:27.164 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:27.186+0000 7f2b538f6640 1 -- 172.21.15.86:0/4211902992 shutdown_connections 2024-03-29T11:22:27.164 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:27.186+0000 7f2b538f6640 1 -- 172.21.15.86:0/4211902992 wait complete. 2024-03-29T11:22:27.164 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:27.186+0000 7f2b538f6640 1 librados: shutdown 2024-03-29T11:22:27.181 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:33.988 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:34.137 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.160+0000 7fdcaeee6640 10 monclient: get_monmap_and_config 2024-03-29T11:22:34.137 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.160+0000 7fdcaeee6640 10 monclient: build_initial_monmap 2024-03-29T11:22:34.138 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.160+0000 7fdcaeee6640 10 monclient: monmap: 2024-03-29T11:22:34.138 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:22:34.138 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:34.138 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:22:34.161068+0000 2024-03-29T11:22:34.138 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:22:34.161068+0000 2024-03-29T11:22:34.138 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:34.138 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:34.138 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:22:34.138 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:22:34.138 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:22:34.138 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:34.141 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.163+0000 7fdcaeee6640 1 Processor -- start 2024-03-29T11:22:34.141 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.164+0000 7fdcaeee6640 1 -- start start 2024-03-29T11:22:34.141 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.164+0000 7fdcaeee6640 10 monclient: init 2024-03-29T11:22:34.141 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.165+0000 7fdcaeee6640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:34.141 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.165+0000 7fdcaeee6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:22:34.142 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.165+0000 7fdcaeee6640 10 monclient(hunting): picked mon.noname-a con 0x7fdca806d9d0 addr v1:172.21.15.86:6789/0 2024-03-29T11:22:34.142 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.165+0000 7fdcaeee6640 10 monclient(hunting): picked mon.noname-c con 0x7fdca813e820 addr v1:172.21.15.196:6789/0 2024-03-29T11:22:34.142 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.165+0000 7fdcaeee6640 10 monclient(hunting): picked mon.noname-b con 0x7fdca80af1d0 addr v1:172.21.15.86:6790/0 2024-03-29T11:22:34.142 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.165+0000 7fdcaeee6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca80aed10 con 0x7fdca806d9d0 2024-03-29T11:22:34.142 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.165+0000 7fdcaeee6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca80afd50 con 0x7fdca813e820 2024-03-29T11:22:34.142 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.165+0000 7fdcaeee6640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca80ab490 con 0x7fdca80af1d0 2024-03-29T11:22:34.142 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.165+0000 7fdcaeee6640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:34.142 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.165+0000 7fdcaeee6640 10 monclient(hunting): authenticate will time out at 978.483676s 2024-03-29T11:22:34.146 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.169+0000 7fdcad483640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7fdca80af1d0 0x7fdca80af640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:52661/0 (socket says 172.21.15.86:52661) 2024-03-29T11:22:34.146 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.169+0000 7fdcad483640 1 -- 172.21.15.86:0/3651219615 learned_addr learned my addr 172.21.15.86:0/3651219615 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:22:34.146 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.170+0000 7fdca77fe640 1 -- 172.21.15.86:0/3651219615 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3923633315 0 0) 0x7fdca80ab490 con 0x7fdca80af1d0 2024-03-29T11:22:34.147 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.170+0000 7fdca77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:34.147 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.170+0000 7fdca77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:34.147 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.170+0000 7fdca77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:34.147 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.170+0000 7fdca77fe640 10 monclient(hunting): my global_id is 4310 2024-03-29T11:22:34.147 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.170+0000 7fdca77fe640 1 -- 172.21.15.86:0/3651219615 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc90003520 con 0x7fdca80af1d0 2024-03-29T11:22:34.147 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.170+0000 7fdca77fe640 1 -- 172.21.15.86:0/3651219615 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1635390152 0 0) 0x7fdc90003520 con 0x7fdca80af1d0 2024-03-29T11:22:34.147 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.170+0000 7fdca77fe640 1 -- 172.21.15.86:0/3651219615 >> v1:172.21.15.196:6789/0 conn(0x7fdca813e820 legacy=0x7fdca813ec50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:34.147 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.171+0000 7fdca77fe640 1 -- 172.21.15.86:0/3651219615 >> v1:172.21.15.86:6789/0 conn(0x7fdca806d9d0 legacy=0x7fdca80b05f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:34.152 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.174+0000 7fdca77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:22:34.152 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.174+0000 7fdca77fe640 1 monclient: found mon.noname-b 2024-03-29T11:22:34.152 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.174+0000 7fdca77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:34.152 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.174+0000 7fdca77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:22:34.152 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.174+0000 7fdca77fe640 1 -- 172.21.15.86:0/3651219615 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca80ab770 con 0x7fdca80af1d0 2024-03-29T11:22:34.152 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.174+0000 7fdca77fe640 10 monclient: _finish_auth 0 2024-03-29T11:22:34.152 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.174+0000 7fdca77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:22:34.153 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.174+0000 7fdca77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:34.153 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.174+0000 7fdca77fe640 1 -- 172.21.15.86:0/3651219615 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fdc98001690 con 0x7fdca80af1d0 2024-03-29T11:22:34.153 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.174+0000 7fdca77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:34.153 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.174+0000 7fdca77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:34.153 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.174+0000 7fdca77fe640 10 monclient: dump: 2024-03-29T11:22:34.153 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:22:34.153 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:34.153 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:34.153 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:34.153 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:34.153 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:34.153 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:22:34.153 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:22:34.153 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:22:34.154 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:34.154 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.175+0000 7fdca77fe640 1 -- 172.21.15.86:0/3651219615 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdc980019a0 con 0x7fdca80af1d0 2024-03-29T11:22:34.154 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.175+0000 7fdca77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:34.154 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.175+0000 7fdca77fe640 1 -- 172.21.15.86:0/3651219615 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fdc98001d30 con 0x7fdca80af1d0 2024-03-29T11:22:34.154 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.175+0000 7fdca77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:34.154 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.175+0000 7fdca77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:34.154 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.175+0000 7fdca77fe640 10 monclient: dump: 2024-03-29T11:22:34.154 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:22:34.154 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:34.154 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:34.154 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:34.154 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:34.154 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:34.154 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:22:34.155 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:22:34.155 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:22:34.155 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:34.156 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.175+0000 7fdcaeee6640 5 monclient: authenticate success, global_id 4310 2024-03-29T11:22:34.156 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.175+0000 7fdcaeee6640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:34.157 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.175+0000 7fdcaeee6640 4 set_mon_vals no callback set 2024-03-29T11:22:34.157 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.176+0000 7fdcaeee6640 10 monclient: shutdown 2024-03-29T11:22:34.157 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.176+0000 7fdcaeee6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:34.157 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.176+0000 7fdcaeee6640 1 -- 172.21.15.86:0/3651219615 >> v1:172.21.15.86:6790/0 conn(0x7fdca80af1d0 legacy=0x7fdca80af640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:34.157 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.176+0000 7fdcaeee6640 1 -- 172.21.15.86:0/3651219615 shutdown_connections 2024-03-29T11:22:34.157 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.176+0000 7fdcaeee6640 1 -- 172.21.15.86:0/3651219615 >> 172.21.15.86:0/3651219615 conn(0x7fdca806b660 msgr2=0x7fdca806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:34.157 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.177+0000 7fdcaeee6640 1 -- 172.21.15.86:0/3651219615 shutdown_connections 2024-03-29T11:22:34.157 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.177+0000 7fdcaeee6640 1 -- 172.21.15.86:0/3651219615 wait complete. 2024-03-29T11:22:34.157 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.177+0000 7fdcaeee6640 10 monclient: build_initial_monmap 2024-03-29T11:22:34.157 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.177+0000 7fdcaeee6640 10 monclient: monmap: 2024-03-29T11:22:34.157 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:22:34.157 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:34.157 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:22:34.158 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:22:34.158 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:34.158 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:34.158 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:22:34.158 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:22:34.158 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:22:34.158 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:34.158 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.179+0000 7fdcaeee6640 1 librados: starting msgr at 2024-03-29T11:22:34.158 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.179+0000 7fdcaeee6640 1 librados: starting objecter 2024-03-29T11:22:34.158 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.179+0000 7fdcaeee6640 1 Processor -- start 2024-03-29T11:22:34.160 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.181+0000 7fdcaeee6640 1 -- start start 2024-03-29T11:22:34.160 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.181+0000 7fdcaeee6640 1 librados: setting wanted keys 2024-03-29T11:22:34.160 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.181+0000 7fdcaeee6640 1 librados: calling monclient init 2024-03-29T11:22:34.160 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.181+0000 7fdcaeee6640 10 monclient: init 2024-03-29T11:22:34.160 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.182+0000 7fdcaeee6640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:34.160 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.182+0000 7fdcaeee6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:22:34.160 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.182+0000 7fdcaeee6640 10 monclient(hunting): picked mon.noname-c con 0x7fdca806d9d0 addr v1:172.21.15.86:6790/0 2024-03-29T11:22:34.160 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.182+0000 7fdcaeee6640 10 monclient(hunting): picked mon.noname-b con 0x7fdca80af1d0 addr v1:172.21.15.196:6789/0 2024-03-29T11:22:34.160 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.182+0000 7fdcaeee6640 10 monclient(hunting): picked mon.noname-a con 0x7fdca813e820 addr v1:172.21.15.86:6789/0 2024-03-29T11:22:34.161 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.182+0000 7fdcaeee6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca80b39c0 con 0x7fdca813e820 2024-03-29T11:22:34.161 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.182+0000 7fdcaeee6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca80b3de0 con 0x7fdca80af1d0 2024-03-29T11:22:34.161 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.182+0000 7fdcaeee6640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdca8137cf0 con 0x7fdca806d9d0 2024-03-29T11:22:34.161 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.182+0000 7fdcaeee6640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:34.161 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.182+0000 7fdcaeee6640 10 monclient(hunting): authenticate will time out at 978.500728s 2024-03-29T11:22:34.171 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.190+0000 7fdcacc82640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7fdca806d9d0 0x7fdca80b2090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:52783/0 (socket says 172.21.15.86:52783) 2024-03-29T11:22:34.171 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.190+0000 7fdcacc82640 1 -- 172.21.15.86:0/612551748 learned_addr learned my addr 172.21.15.86:0/612551748 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:22:34.171 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.191+0000 7fdca57fa640 10 client.?.objecter ms_handle_connect 0x7fdca806d9d0 2024-03-29T11:22:34.172 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.191+0000 7fdca57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:34.172 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.191+0000 7fdca57fa640 1 -- 172.21.15.86:0/612551748 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4049741916 0 0) 0x7fdca8137cf0 con 0x7fdca806d9d0 2024-03-29T11:22:34.172 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.191+0000 7fdca57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:22:34.172 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.191+0000 7fdca57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:34.172 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.191+0000 7fdca57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:34.172 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.191+0000 7fdca57fa640 10 monclient(hunting): my global_id is 4313 2024-03-29T11:22:34.172 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.191+0000 7fdca57fa640 1 -- 172.21.15.86:0/612551748 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdc94004eb0 con 0x7fdca806d9d0 2024-03-29T11:22:34.172 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.192+0000 7fdca57fa640 1 -- 172.21.15.86:0/612551748 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 968720938 0 0) 0x7fdc94004eb0 con 0x7fdca806d9d0 2024-03-29T11:22:34.172 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.192+0000 7fdca57fa640 1 -- 172.21.15.86:0/612551748 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fdca8137cf0 con 0x7fdca806d9d0 2024-03-29T11:22:34.172 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.192+0000 7fdca57fa640 1 -- 172.21.15.86:0/612551748 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fdca0001b50 con 0x7fdca806d9d0 2024-03-29T11:22:34.172 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.192+0000 7fdca57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:34.172 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.192+0000 7fdca57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:34.172 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.192+0000 7fdca57fa640 10 monclient(hunting): dump: 2024-03-29T11:22:34.172 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:22:34.172 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:34.173 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:34.173 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:34.173 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:34.173 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:34.173 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:22:34.173 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:22:34.173 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:22:34.173 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:34.173 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.192+0000 7fdca57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:34.173 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.193+0000 7fdca57fa640 10 client.?.objecter ms_handle_connect 0x7fdca813e820 2024-03-29T11:22:34.173 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.193+0000 7fdca57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:34.173 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.193+0000 7fdca57fa640 1 -- 172.21.15.86:0/612551748 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1757923485 0 0) 0x7fdca8137cf0 con 0x7fdca806d9d0 2024-03-29T11:22:34.173 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.193+0000 7fdca57fa640 1 -- 172.21.15.86:0/612551748 >> v1:172.21.15.196:6789/0 conn(0x7fdca80af1d0 legacy=0x7fdca81375e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:34.173 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.194+0000 7fdca57fa640 1 -- 172.21.15.86:0/612551748 >> v1:172.21.15.86:6789/0 conn(0x7fdca813e820 legacy=0x7fdca81e1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:34.173 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.194+0000 7fdca57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:22:34.173 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.194+0000 7fdca57fa640 1 monclient: found mon.c 2024-03-29T11:22:34.174 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.194+0000 7fdca57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:34.174 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.194+0000 7fdca57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:22:34.174 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.194+0000 7fdca57fa640 1 -- 172.21.15.86:0/612551748 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca80b42c0 con 0x7fdca806d9d0 2024-03-29T11:22:34.174 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.194+0000 7fdca57fa640 10 monclient: _finish_auth 0 2024-03-29T11:22:34.174 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.194+0000 7fdca57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:22:34.174 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.194+0000 7fdca57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:34.174 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.194+0000 7fdcaeee6640 5 monclient: authenticate success, global_id 4313 2024-03-29T11:22:34.174 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.194+0000 7fdcaeee6640 10 monclient: _renew_subs 2024-03-29T11:22:34.174 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.194+0000 7fdcaeee6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:22:34.174 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.194+0000 7fdcaeee6640 1 -- 172.21.15.86:0/612551748 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdca80b1380 con 0x7fdca806d9d0 2024-03-29T11:22:34.174 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.194+0000 7fdcaeee6640 10 client.4313.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:34.174 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.194+0000 7fdcaeee6640 10 monclient: _renew_subs 2024-03-29T11:22:34.175 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.194+0000 7fdcaeee6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:22:34.175 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.194+0000 7fdcaeee6640 1 -- 172.21.15.86:0/612551748 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fdca80b2980 con 0x7fdca806d9d0 2024-03-29T11:22:34.175 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.194+0000 7fdcaeee6640 1 librados: init done 2024-03-29T11:22:34.175 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.197+0000 7fdcaeee6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:34.175 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.197+0000 7fdcaeee6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:34.175 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.197+0000 7fdcaeee6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:22:34.177 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.197+0000 7fdcaeee6640 1 -- 172.21.15.86:0/612551748 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdca80b37e0 con 0x7fdca806d9d0 2024-03-29T11:22:34.177 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.199+0000 7fdca57fa640 1 -- 172.21.15.86:0/612551748 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fdca0001cb0 con 0x7fdca806d9d0 2024-03-29T11:22:34.177 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.199+0000 7fdca57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:34.177 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.199+0000 7fdca57fa640 1 -- 172.21.15.86:0/612551748 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fdca0004dc0 con 0x7fdca806d9d0 2024-03-29T11:22:34.177 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.199+0000 7fdca57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:34.177 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.199+0000 7fdca57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:34.177 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.199+0000 7fdca57fa640 10 monclient: dump: 2024-03-29T11:22:34.177 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:22:34.177 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:34.178 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:34.178 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:34.178 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:34.178 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:34.178 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:22:34.178 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:22:34.178 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:22:34.178 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:34.178 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.199+0000 7fdca57fa640 1 -- 172.21.15.86:0/612551748 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7fdca001ae50 con 0x7fdca806d9d0 2024-03-29T11:22:34.178 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.200+0000 7fdca67fc640 4 set_mon_vals no callback set 2024-03-29T11:22:34.179 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.201+0000 7fdca7fff640 10 monclient: get_auth_request con 0x7fdc9406dbd0 auth_method 0 2024-03-29T11:22:34.179 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.201+0000 7fdca57fa640 1 -- 172.21.15.86:0/612551748 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7fdca0084f40 con 0x7fdca806d9d0 2024-03-29T11:22:34.179 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.201+0000 7fdca57fa640 10 client.4313.objecter ms_dispatch 0x7fdca8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:34.179 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.201+0000 7fdca57fa640 3 client.4313.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:34.180 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.201+0000 7fdca57fa640 3 client.4313.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:34.180 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.202+0000 7fdca57fa640 20 client.4313.objecter dump_active .. 0 homeless 2024-03-29T11:22:34.180 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.202+0000 7fdca57fa640 10 client.4313.objecter ms_handle_connect 0x7fdc9406dbd0 2024-03-29T11:22:34.184 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.208+0000 7fdca57fa640 1 -- 172.21.15.86:0/612551748 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fdca00506d0 con 0x7fdca806d9d0 2024-03-29T11:22:34.184 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.208+0000 7fdca57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:34.184 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.208+0000 7fdca57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:34.507 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.530+0000 7fdcaeee6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:34.507 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.530+0000 7fdcaeee6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:34.507 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.530+0000 7fdcaeee6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:22:34.507 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.530+0000 7fdcaeee6640 1 -- 172.21.15.86:0/612551748 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdca8075f10 con 0x7fdca806d9d0 2024-03-29T11:22:34.517 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.540+0000 7fdca57fa640 1 -- 172.21.15.86:0/612551748 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7fdca00527d0 con 0x7fdca806d9d0 2024-03-29T11:22:34.517 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.540+0000 7fdca57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:34.517 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.540+0000 7fdca57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:34.517 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:22:34.518 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:34.523 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.544+0000 7fdcaeee6640 10 librados: watch_flush enter 2024-03-29T11:22:34.523 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.544+0000 7fdcaeee6640 10 librados: watch_flush exit 2024-03-29T11:22:34.523 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.544+0000 7fdcaeee6640 20 client.4313.objecter shutdown clearing up homeless session... 2024-03-29T11:22:34.524 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.544+0000 7fdcaeee6640 10 client.4313.objecter successfully canceled tick 2024-03-29T11:22:34.524 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.544+0000 7fdcaeee6640 1 -- 172.21.15.86:0/612551748 >> v1:172.21.15.196:6816/588110318 conn(0x7fdc9406dbd0 legacy=0x7fdc94070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:34.524 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.544+0000 7fdcaeee6640 10 monclient: shutdown 2024-03-29T11:22:34.524 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.544+0000 7fdcaeee6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:34.524 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.544+0000 7fdcaeee6640 1 -- 172.21.15.86:0/612551748 >> v1:172.21.15.86:6790/0 conn(0x7fdca806d9d0 legacy=0x7fdca80b2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:34.524 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.547+0000 7fdcaeee6640 1 -- 172.21.15.86:0/612551748 shutdown_connections 2024-03-29T11:22:34.524 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.547+0000 7fdcaeee6640 1 -- 172.21.15.86:0/612551748 >> 172.21.15.86:0/612551748 conn(0x7fdca806b660 msgr2=0x7fdca814d460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:34.524 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.547+0000 7fdcaeee6640 1 -- 172.21.15.86:0/612551748 shutdown_connections 2024-03-29T11:22:34.524 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.547+0000 7fdcaeee6640 1 -- 172.21.15.86:0/612551748 wait complete. 2024-03-29T11:22:34.524 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:34.547+0000 7fdcaeee6640 1 librados: shutdown 2024-03-29T11:22:34.544 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:41.352 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:41.504 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.527+0000 7f6ab86f1640 10 monclient: get_monmap_and_config 2024-03-29T11:22:41.504 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.527+0000 7f6ab86f1640 10 monclient: build_initial_monmap 2024-03-29T11:22:41.504 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.527+0000 7f6ab86f1640 10 monclient: monmap: 2024-03-29T11:22:41.504 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:22:41.504 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:41.504 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:22:41.528331+0000 2024-03-29T11:22:41.504 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:22:41.528331+0000 2024-03-29T11:22:41.504 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:41.505 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:41.505 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:22:41.505 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:22:41.505 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:22:41.505 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:41.507 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.530+0000 7f6ab86f1640 1 Processor -- start 2024-03-29T11:22:41.507 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.530+0000 7f6ab86f1640 1 -- start start 2024-03-29T11:22:41.507 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.530+0000 7f6ab86f1640 10 monclient: init 2024-03-29T11:22:41.509 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.531+0000 7f6ab86f1640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:41.509 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.531+0000 7f6ab86f1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:22:41.509 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.531+0000 7f6ab86f1640 10 monclient(hunting): picked mon.noname-a con 0x7f6ab00b4330 addr v1:172.21.15.86:6789/0 2024-03-29T11:22:41.509 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.531+0000 7f6ab86f1640 10 monclient(hunting): picked mon.noname-b con 0x7f6ab01470b0 addr v1:172.21.15.86:6790/0 2024-03-29T11:22:41.509 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.531+0000 7f6ab86f1640 10 monclient(hunting): picked mon.noname-c con 0x7f6ab01439b0 addr v1:172.21.15.196:6789/0 2024-03-29T11:22:41.509 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.531+0000 7f6ab86f1640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ab006f2a0 con 0x7f6ab00b4330 2024-03-29T11:22:41.509 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.531+0000 7f6ab86f1640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ab0146b90 con 0x7f6ab01439b0 2024-03-29T11:22:41.509 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.531+0000 7f6ab86f1640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ab00b2ca0 con 0x7f6ab01470b0 2024-03-29T11:22:41.510 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.531+0000 7f6ab86f1640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:41.510 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.531+0000 7f6ab86f1640 10 monclient(hunting): authenticate will time out at 985.849355s 2024-03-29T11:22:41.512 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.534+0000 7f6ab6c8e640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f6ab01439b0 0x7f6ab0151d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:50205/0 (socket says 172.21.15.86:50205) 2024-03-29T11:22:41.512 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.534+0000 7f6ab6c8e640 1 -- 172.21.15.86:0/285611937 learned_addr learned my addr 172.21.15.86:0/285611937 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:22:41.526 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.536+0000 7f6ab548b640 1 -- 172.21.15.86:0/285611937 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 604194208 0 0) 0x7f6ab0146b90 con 0x7f6ab01439b0 2024-03-29T11:22:41.526 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.536+0000 7f6ab548b640 10 monclient(hunting): _negotiate 2024-03-29T11:22:41.526 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.536+0000 7f6ab548b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:41.527 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.536+0000 7f6ab548b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:41.527 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.536+0000 7f6ab548b640 10 monclient(hunting): my global_id is 4285 2024-03-29T11:22:41.527 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.536+0000 7f6ab548b640 1 -- 172.21.15.86:0/285611937 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a9c003520 con 0x7f6ab01439b0 2024-03-29T11:22:41.527 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.536+0000 7f6ab548b640 1 -- 172.21.15.86:0/285611937 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2411983253 0 0) 0x7f6a9c003520 con 0x7f6ab01439b0 2024-03-29T11:22:41.527 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.536+0000 7f6ab548b640 1 -- 172.21.15.86:0/285611937 >> v1:172.21.15.86:6790/0 conn(0x7f6ab01470b0 legacy=0x7f6ab00b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:41.527 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.537+0000 7f6ab548b640 1 -- 172.21.15.86:0/285611937 >> v1:172.21.15.86:6789/0 conn(0x7f6ab00b4330 legacy=0x7f6ab00b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:41.528 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.537+0000 7f6ab548b640 10 monclient: _finish_hunting 0 2024-03-29T11:22:41.528 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.537+0000 7f6ab548b640 1 monclient: found mon.noname-c 2024-03-29T11:22:41.528 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.537+0000 7f6ab548b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:41.528 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.537+0000 7f6ab548b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T11:22:41.528 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.537+0000 7f6ab548b640 1 -- 172.21.15.86:0/285611937 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab006e0c0 con 0x7f6ab01439b0 2024-03-29T11:22:41.528 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.538+0000 7f6ab548b640 10 monclient: _finish_auth 0 2024-03-29T11:22:41.528 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.538+0000 7f6ab548b640 10 monclient: _check_auth_tickets 2024-03-29T11:22:41.529 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.538+0000 7f6ab548b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:41.529 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.538+0000 7f6ab86f1640 5 monclient: authenticate success, global_id 4285 2024-03-29T11:22:41.529 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.538+0000 7f6ab86f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:41.529 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.538+0000 7f6ab548b640 1 -- 172.21.15.86:0/285611937 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f6aa8001a10 con 0x7f6ab01439b0 2024-03-29T11:22:41.529 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.538+0000 7f6ab548b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:41.529 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.538+0000 7f6ab548b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:41.530 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.538+0000 7f6ab548b640 10 monclient: dump: 2024-03-29T11:22:41.530 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:22:41.530 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:41.530 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:41.530 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:41.530 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:41.531 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:41.531 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:22:41.531 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:22:41.531 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:22:41.531 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:41.531 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.538+0000 7f6ab86f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:41.532 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.538+0000 7f6ab548b640 1 -- 172.21.15.86:0/285611937 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6aa8001ca0 con 0x7f6ab01439b0 2024-03-29T11:22:41.532 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.538+0000 7f6ab548b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:41.532 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.538+0000 7f6ab86f1640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:41.532 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.538+0000 7f6ab86f1640 4 set_mon_vals no callback set 2024-03-29T11:22:41.532 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.539+0000 7f6ab548b640 1 -- 172.21.15.86:0/285611937 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f6aa8002030 con 0x7f6ab01439b0 2024-03-29T11:22:41.532 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.539+0000 7f6ab548b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:41.533 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.539+0000 7f6ab548b640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:41.533 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.539+0000 7f6ab548b640 10 monclient: dump: 2024-03-29T11:22:41.533 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:22:41.533 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:41.533 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:41.533 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:41.533 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:41.534 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:41.534 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:22:41.534 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:22:41.534 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:22:41.534 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:41.535 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.539+0000 7f6ab86f1640 10 monclient: shutdown 2024-03-29T11:22:41.535 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.539+0000 7f6ab86f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:41.535 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.539+0000 7f6ab86f1640 1 -- 172.21.15.86:0/285611937 >> v1:172.21.15.196:6789/0 conn(0x7f6ab01439b0 legacy=0x7f6ab0151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:41.535 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.539+0000 7f6ab86f1640 1 -- 172.21.15.86:0/285611937 shutdown_connections 2024-03-29T11:22:41.535 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.539+0000 7f6ab86f1640 1 -- 172.21.15.86:0/285611937 >> 172.21.15.86:0/285611937 conn(0x7f6ab0066a00 msgr2=0x7f6ab0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:41.535 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.539+0000 7f6ab86f1640 1 -- 172.21.15.86:0/285611937 shutdown_connections 2024-03-29T11:22:41.535 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.539+0000 7f6ab86f1640 1 -- 172.21.15.86:0/285611937 wait complete. 2024-03-29T11:22:41.536 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.540+0000 7f6ab86f1640 10 monclient: build_initial_monmap 2024-03-29T11:22:41.536 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.540+0000 7f6ab86f1640 10 monclient: monmap: 2024-03-29T11:22:41.536 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:22:41.536 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:41.536 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:22:41.537 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:22:41.537 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:41.537 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:41.537 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:22:41.537 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:22:41.537 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:22:41.538 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:41.538 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.541+0000 7f6ab86f1640 1 librados: starting msgr at 2024-03-29T11:22:41.538 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.541+0000 7f6ab86f1640 1 librados: starting objecter 2024-03-29T11:22:41.538 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.541+0000 7f6ab86f1640 1 Processor -- start 2024-03-29T11:22:41.538 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.541+0000 7f6ab86f1640 1 -- start start 2024-03-29T11:22:41.538 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.541+0000 7f6ab86f1640 1 librados: setting wanted keys 2024-03-29T11:22:41.539 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.541+0000 7f6ab86f1640 1 librados: calling monclient init 2024-03-29T11:22:41.539 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.541+0000 7f6ab86f1640 10 monclient: init 2024-03-29T11:22:41.539 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.542+0000 7f6ab86f1640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:41.539 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.542+0000 7f6ab86f1640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:22:41.539 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.542+0000 7f6ab86f1640 10 monclient(hunting): picked mon.noname-b con 0x7f6ab00b4330 addr v1:172.21.15.196:6789/0 2024-03-29T11:22:41.539 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.542+0000 7f6ab86f1640 10 monclient(hunting): picked mon.noname-a con 0x7f6ab01439b0 addr v1:172.21.15.86:6789/0 2024-03-29T11:22:41.539 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.542+0000 7f6ab86f1640 10 monclient(hunting): picked mon.noname-c con 0x7f6ab01470b0 addr v1:172.21.15.86:6790/0 2024-03-29T11:22:41.540 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.542+0000 7f6ab86f1640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ab01466b0 con 0x7f6ab01439b0 2024-03-29T11:22:41.540 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.542+0000 7f6ab86f1640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ab01213a0 con 0x7f6ab00b4330 2024-03-29T11:22:41.540 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.542+0000 7f6ab86f1640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ab0144350 con 0x7f6ab01470b0 2024-03-29T11:22:41.540 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.542+0000 7f6ab86f1640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:41.540 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.542+0000 7f6ab86f1640 10 monclient(hunting): authenticate will time out at 985.860296s 2024-03-29T11:22:41.540 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.545+0000 7f6ab6c8e640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f6ab01470b0 0x7f6ab0141430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:36441/0 (socket says 172.21.15.86:36441) 2024-03-29T11:22:41.541 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.545+0000 7f6ab6c8e640 1 -- 172.21.15.86:0/1888034630 learned_addr learned my addr 172.21.15.86:0/1888034630 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:22:41.541 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.548+0000 7f6aa6ffd640 10 client.?.objecter ms_handle_connect 0x7f6ab01470b0 2024-03-29T11:22:41.541 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.548+0000 7f6aa6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:41.541 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.549+0000 7f6aa6ffd640 1 -- 172.21.15.86:0/1888034630 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 258790714 0 0) 0x7f6ab0144350 con 0x7f6ab01470b0 2024-03-29T11:22:41.541 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.549+0000 7f6aa6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:41.541 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.549+0000 7f6aa6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:41.542 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.549+0000 7f6aa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:41.542 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.549+0000 7f6aa6ffd640 10 monclient(hunting): my global_id is 4316 2024-03-29T11:22:41.542 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.549+0000 7f6aa6ffd640 1 -- 172.21.15.86:0/1888034630 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a94004eb0 con 0x7f6ab01470b0 2024-03-29T11:22:41.542 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.549+0000 7f6aa6ffd640 10 client.?.objecter ms_handle_connect 0x7f6ab00b4330 2024-03-29T11:22:41.542 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.549+0000 7f6aa6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:41.544 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.550+0000 7f6aa6ffd640 1 -- 172.21.15.86:0/1888034630 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1872590367 0 0) 0x7f6a94004eb0 con 0x7f6ab01470b0 2024-03-29T11:22:41.545 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.550+0000 7f6aa6ffd640 1 -- 172.21.15.86:0/1888034630 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6ab0144350 con 0x7f6ab01470b0 2024-03-29T11:22:41.545 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.550+0000 7f6aa6ffd640 1 -- 172.21.15.86:0/1888034630 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f6aa8001990 con 0x7f6ab01470b0 2024-03-29T11:22:41.545 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.550+0000 7f6aa6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:41.545 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.550+0000 7f6aa6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:41.545 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.550+0000 7f6aa6ffd640 10 monclient(hunting): dump: 2024-03-29T11:22:41.545 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:22:41.545 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:41.545 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:41.545 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:41.545 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:41.546 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:41.546 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:22:41.546 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:22:41.546 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:22:41.546 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:41.546 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.550+0000 7f6aa6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:41.546 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.550+0000 7f6aa6ffd640 1 -- 172.21.15.86:0/1888034630 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1893207523 0 0) 0x7f6ab0144350 con 0x7f6ab01470b0 2024-03-29T11:22:41.546 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.551+0000 7f6aa6ffd640 1 -- 172.21.15.86:0/1888034630 >> v1:172.21.15.196:6789/0 conn(0x7f6ab00b4330 legacy=0x7f6ab011fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:41.546 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.551+0000 7f6aa6ffd640 1 -- 172.21.15.86:0/1888034630 >> v1:172.21.15.86:6789/0 conn(0x7f6ab01439b0 legacy=0x7f6ab0140d20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:22:41.546 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.553+0000 7f6aa6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:22:41.547 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.553+0000 7f6aa6ffd640 1 monclient: found mon.c 2024-03-29T11:22:41.547 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.553+0000 7f6aa6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:41.547 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.553+0000 7f6aa6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:22:41.547 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.553+0000 7f6aa6ffd640 1 -- 172.21.15.86:0/1888034630 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ab0141b40 con 0x7f6ab01470b0 2024-03-29T11:22:41.547 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.553+0000 7f6aa6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:22:41.547 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.553+0000 7f6aa6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:22:41.547 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.553+0000 7f6aa6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:41.547 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.553+0000 7f6aa6ffd640 1 -- 172.21.15.86:0/1888034630 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6aa8004e10 con 0x7f6ab01470b0 2024-03-29T11:22:41.547 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.553+0000 7f6aa6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:41.547 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.553+0000 7f6aa6ffd640 1 -- 172.21.15.86:0/1888034630 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f6aa8005160 con 0x7f6ab01470b0 2024-03-29T11:22:41.548 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.553+0000 7f6aa6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:41.548 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.553+0000 7f6aa6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:41.548 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.553+0000 7f6aa6ffd640 10 monclient: dump: 2024-03-29T11:22:41.548 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:22:41.548 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:41.548 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:41.548 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:41.548 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:41.548 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:41.548 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:22:41.549 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:22:41.549 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:22:41.549 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:41.549 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.558+0000 7f6ab86f1640 5 monclient: authenticate success, global_id 4316 2024-03-29T11:22:41.549 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.558+0000 7f6ab86f1640 10 monclient: _renew_subs 2024-03-29T11:22:41.549 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.558+0000 7f6ab86f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:22:41.549 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.558+0000 7f6ab86f1640 1 -- 172.21.15.86:0/1888034630 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ab0146090 con 0x7f6ab01470b0 2024-03-29T11:22:41.549 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.558+0000 7f6ab86f1640 10 client.4316.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:41.549 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.558+0000 7f6ab86f1640 10 monclient: _renew_subs 2024-03-29T11:22:41.549 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.558+0000 7f6ab86f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:22:41.550 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.558+0000 7f6ab86f1640 1 -- 172.21.15.86:0/1888034630 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ab0141d80 con 0x7f6ab01470b0 2024-03-29T11:22:41.550 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.558+0000 7f6ab86f1640 1 librados: init done 2024-03-29T11:22:41.550 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.558+0000 7f6ab4c8a640 4 set_mon_vals no callback set 2024-03-29T11:22:41.550 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.562+0000 7f6ab86f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:41.550 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.562+0000 7f6ab86f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:41.550 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.562+0000 7f6ab86f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:22:41.550 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.562+0000 7f6ab86f1640 1 -- 172.21.15.86:0/1888034630 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a98005080 con 0x7f6ab01470b0 2024-03-29T11:22:41.550 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.564+0000 7f6aa6ffd640 1 -- 172.21.15.86:0/1888034630 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f6aa801b130 con 0x7f6ab01470b0 2024-03-29T11:22:41.550 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.564+0000 7f6aa6ffd640 1 -- 172.21.15.86:0/1888034630 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f6aa8084f10 con 0x7f6ab01470b0 2024-03-29T11:22:41.550 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.564+0000 7f6aa6ffd640 10 client.4316.objecter ms_dispatch 0x7f6ab0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:41.551 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.564+0000 7f6aa6ffd640 3 client.4316.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:41.551 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.564+0000 7f6aa6ffd640 3 client.4316.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:41.551 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.564+0000 7f6aa6ffd640 20 client.4316.objecter dump_active .. 0 homeless 2024-03-29T11:22:41.551 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.564+0000 7f6ab648d640 10 monclient: get_auth_request con 0x7f6a9406db70 auth_method 0 2024-03-29T11:22:41.551 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.565+0000 7f6aa6ffd640 10 client.4316.objecter ms_handle_connect 0x7f6a9406db70 2024-03-29T11:22:41.554 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.577+0000 7f6aa6ffd640 1 -- 172.21.15.86:0/1888034630 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6aa80507f0 con 0x7f6ab01470b0 2024-03-29T11:22:41.554 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.577+0000 7f6aa6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:41.554 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.577+0000 7f6aa6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:41.887 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.909+0000 7f6ab86f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:41.888 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.909+0000 7f6ab86f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:41.888 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.909+0000 7f6ab86f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:22:41.888 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.909+0000 7f6ab86f1640 1 -- 172.21.15.86:0/1888034630 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a98005330 con 0x7f6ab01470b0 2024-03-29T11:22:41.889 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.912+0000 7f6aa6ffd640 1 -- 172.21.15.86:0/1888034630 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f6aa8052720 con 0x7f6ab01470b0 2024-03-29T11:22:41.889 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.912+0000 7f6aa6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:41.889 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.912+0000 7f6aa6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:41.890 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:22:41.890 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:41.894 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.917+0000 7f6ab86f1640 10 librados: watch_flush enter 2024-03-29T11:22:41.896 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.918+0000 7f6ab86f1640 10 librados: watch_flush exit 2024-03-29T11:22:41.896 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.918+0000 7f6ab86f1640 20 client.4316.objecter shutdown clearing up homeless session... 2024-03-29T11:22:41.896 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.918+0000 7f6ab86f1640 10 client.4316.objecter successfully canceled tick 2024-03-29T11:22:41.896 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.919+0000 7f6ab86f1640 1 -- 172.21.15.86:0/1888034630 >> v1:172.21.15.196:6816/588110318 conn(0x7f6a9406db70 legacy=0x7f6a94070030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:41.896 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.919+0000 7f6ab86f1640 10 monclient: shutdown 2024-03-29T11:22:41.896 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.919+0000 7f6ab86f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:41.896 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.919+0000 7f6ab86f1640 1 -- 172.21.15.86:0/1888034630 >> v1:172.21.15.86:6790/0 conn(0x7f6ab01470b0 legacy=0x7f6ab0141430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:41.897 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.920+0000 7f6ab86f1640 1 -- 172.21.15.86:0/1888034630 shutdown_connections 2024-03-29T11:22:41.897 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.920+0000 7f6ab86f1640 1 -- 172.21.15.86:0/1888034630 >> 172.21.15.86:0/1888034630 conn(0x7f6ab0066a00 msgr2=0x7f6ab0070dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:41.898 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.921+0000 7f6ab86f1640 1 -- 172.21.15.86:0/1888034630 shutdown_connections 2024-03-29T11:22:41.898 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.921+0000 7f6ab86f1640 1 -- 172.21.15.86:0/1888034630 wait complete. 2024-03-29T11:22:41.899 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:41.922+0000 7f6ab86f1640 1 librados: shutdown 2024-03-29T11:22:41.919 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:48.726 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:48.882 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.904+0000 7f64a07fe640 10 monclient: get_monmap_and_config 2024-03-29T11:22:48.883 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.904+0000 7f64a07fe640 10 monclient: build_initial_monmap 2024-03-29T11:22:48.883 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.904+0000 7f64a07fe640 10 monclient: monmap: 2024-03-29T11:22:48.883 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:22:48.883 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:48.883 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:22:48.905507+0000 2024-03-29T11:22:48.883 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:22:48.905507+0000 2024-03-29T11:22:48.883 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:48.883 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:48.883 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:22:48.883 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:22:48.883 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:22:48.883 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:48.884 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.905+0000 7f64a07fe640 1 Processor -- start 2024-03-29T11:22:48.884 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.905+0000 7f64a07fe640 1 -- start start 2024-03-29T11:22:48.884 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.905+0000 7f64a07fe640 10 monclient: init 2024-03-29T11:22:48.884 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.906+0000 7f64a07fe640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:48.884 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.906+0000 7f64a07fe640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:22:48.884 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.906+0000 7f64a07fe640 10 monclient(hunting): picked mon.noname-b con 0x7f64980b1f60 addr v1:172.21.15.86:6790/0 2024-03-29T11:22:48.884 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.906+0000 7f64a07fe640 10 monclient(hunting): picked mon.noname-a con 0x7f6498071eb0 addr v1:172.21.15.86:6789/0 2024-03-29T11:22:48.884 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.906+0000 7f64a07fe640 10 monclient(hunting): picked mon.noname-c con 0x7f6498143d10 addr v1:172.21.15.196:6789/0 2024-03-29T11:22:48.884 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.906+0000 7f64a07fe640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f649806ef80 con 0x7f6498071eb0 2024-03-29T11:22:48.884 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.906+0000 7f64a07fe640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f649806f260 con 0x7f6498143d10 2024-03-29T11:22:48.884 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.906+0000 7f64a07fe640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64980b40c0 con 0x7f64980b1f60 2024-03-29T11:22:48.885 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.906+0000 7f64a07fe640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:48.885 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.906+0000 7f64a07fe640 10 monclient(hunting): authenticate will time out at 993.224743s 2024-03-29T11:22:48.890 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.913+0000 7f649ed9b640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f6498143d10 0x7f6498146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:41805/0 (socket says 172.21.15.86:41805) 2024-03-29T11:22:48.890 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.913+0000 7f649ed9b640 1 -- 172.21.15.86:0/1678611761 learned_addr learned my addr 172.21.15.86:0/1678611761 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:22:48.891 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.914+0000 7f649d598640 1 -- 172.21.15.86:0/1678611761 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 574385662 0 0) 0x7f649806f260 con 0x7f6498143d10 2024-03-29T11:22:48.892 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.914+0000 7f649d598640 10 monclient(hunting): _negotiate 2024-03-29T11:22:48.892 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.914+0000 7f649d598640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:48.892 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.914+0000 7f649d598640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:48.892 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.914+0000 7f649d598640 10 monclient(hunting): my global_id is 4291 2024-03-29T11:22:48.892 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.914+0000 7f649d598640 1 -- 172.21.15.86:0/1678611761 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6484003520 con 0x7f6498143d10 2024-03-29T11:22:48.892 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.914+0000 7f649d598640 1 -- 172.21.15.86:0/1678611761 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2583736582 0 0) 0x7f6484003520 con 0x7f6498143d10 2024-03-29T11:22:48.892 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.914+0000 7f649d598640 1 -- 172.21.15.86:0/1678611761 >> v1:172.21.15.86:6790/0 conn(0x7f64980b1f60 legacy=0x7f6498143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:22:48.892 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.914+0000 7f649d598640 1 -- 172.21.15.86:0/1678611761 >> v1:172.21.15.86:6789/0 conn(0x7f6498071eb0 legacy=0x7f6498146470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:22:48.894 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.914+0000 7f649d598640 10 monclient: _finish_hunting 0 2024-03-29T11:22:48.894 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.914+0000 7f649d598640 1 monclient: found mon.noname-c 2024-03-29T11:22:48.894 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.914+0000 7f649d598640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:48.894 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.914+0000 7f649d598640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T11:22:48.894 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.914+0000 7f649d598640 1 -- 172.21.15.86:0/1678611761 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6498155d70 con 0x7f6498143d10 2024-03-29T11:22:48.894 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.914+0000 7f649d598640 10 monclient: _finish_auth 0 2024-03-29T11:22:48.894 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.914+0000 7f649d598640 10 monclient: _check_auth_tickets 2024-03-29T11:22:48.894 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.914+0000 7f649d598640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:48.894 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.914+0000 7f649d598640 1 -- 172.21.15.86:0/1678611761 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f6494001690 con 0x7f6498143d10 2024-03-29T11:22:48.894 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.914+0000 7f649d598640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:48.895 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.914+0000 7f649d598640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:48.895 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.914+0000 7f649d598640 10 monclient: dump: 2024-03-29T11:22:48.895 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:22:48.895 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:48.895 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:48.895 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:48.895 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:48.895 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:48.895 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:22:48.895 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:22:48.895 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:22:48.895 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:48.895 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.915+0000 7f64a07fe640 5 monclient: authenticate success, global_id 4291 2024-03-29T11:22:48.896 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.915+0000 7f64a07fe640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:48.896 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.918+0000 7f649d598640 1 -- 172.21.15.86:0/1678611761 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6494001860 con 0x7f6498143d10 2024-03-29T11:22:48.896 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.918+0000 7f649d598640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:48.896 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.918+0000 7f649d598640 1 -- 172.21.15.86:0/1678611761 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f6494001cb0 con 0x7f6498143d10 2024-03-29T11:22:48.896 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.918+0000 7f649d598640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:48.896 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.918+0000 7f649d598640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:48.896 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.918+0000 7f649d598640 10 monclient: dump: 2024-03-29T11:22:48.896 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:22:48.896 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:48.896 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:48.897 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:48.897 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:48.897 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:48.897 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:22:48.897 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:22:48.897 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:22:48.897 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:48.897 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.919+0000 7f64a07fe640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:48.897 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.919+0000 7f64a07fe640 4 set_mon_vals no callback set 2024-03-29T11:22:48.898 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.920+0000 7f64a07fe640 10 monclient: shutdown 2024-03-29T11:22:48.898 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.920+0000 7f64a07fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:48.898 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.920+0000 7f64a07fe640 1 -- 172.21.15.86:0/1678611761 >> v1:172.21.15.196:6789/0 conn(0x7f6498143d10 legacy=0x7f6498146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:48.898 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.921+0000 7f64a07fe640 1 -- 172.21.15.86:0/1678611761 shutdown_connections 2024-03-29T11:22:48.898 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.921+0000 7f64a07fe640 1 -- 172.21.15.86:0/1678611761 >> 172.21.15.86:0/1678611761 conn(0x7f6498066a00 msgr2=0x7f6498066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:48.898 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.921+0000 7f64a07fe640 1 -- 172.21.15.86:0/1678611761 shutdown_connections 2024-03-29T11:22:48.899 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.921+0000 7f64a07fe640 1 -- 172.21.15.86:0/1678611761 wait complete. 2024-03-29T11:22:48.900 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.922+0000 7f64a07fe640 10 monclient: build_initial_monmap 2024-03-29T11:22:48.900 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.922+0000 7f64a07fe640 10 monclient: monmap: 2024-03-29T11:22:48.901 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:22:48.901 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:48.901 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:22:48.901 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:22:48.901 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:48.901 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:48.901 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:22:48.901 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:22:48.901 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:22:48.901 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:48.901 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.923+0000 7f64a07fe640 1 librados: starting msgr at 2024-03-29T11:22:48.901 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.923+0000 7f64a07fe640 1 librados: starting objecter 2024-03-29T11:22:48.901 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.923+0000 7f64a07fe640 1 Processor -- start 2024-03-29T11:22:48.901 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.923+0000 7f64a07fe640 1 -- start start 2024-03-29T11:22:48.901 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.923+0000 7f64a07fe640 1 librados: setting wanted keys 2024-03-29T11:22:48.902 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.923+0000 7f64a07fe640 1 librados: calling monclient init 2024-03-29T11:22:48.902 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.923+0000 7f64a07fe640 10 monclient: init 2024-03-29T11:22:48.902 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.924+0000 7f64a07fe640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:48.902 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.924+0000 7f64a07fe640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:22:48.904 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.927+0000 7f64a07fe640 10 monclient(hunting): picked mon.noname-a con 0x7f64980b1f60 addr v1:172.21.15.86:6789/0 2024-03-29T11:22:48.904 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.927+0000 7f649e59a640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f64980b1f60 0x7f64981428f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:58121/0 (socket says 172.21.15.86:58121) 2024-03-29T11:22:48.904 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.927+0000 7f649e59a640 1 -- 172.21.15.86:0/3726113586 learned_addr learned my addr 172.21.15.86:0/3726113586 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:22:48.905 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.927+0000 7f648effd640 10 client.?.objecter ms_handle_connect 0x7f64980b1f60 2024-03-29T11:22:48.905 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.928+0000 7f648effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:48.908 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.931+0000 7f64a07fe640 10 monclient(hunting): picked mon.noname-b con 0x7f6498143d10 addr v1:172.21.15.196:6789/0 2024-03-29T11:22:48.910 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.933+0000 7f64a07fe640 10 monclient(hunting): picked mon.noname-c con 0x7f64981542d0 addr v1:172.21.15.86:6790/0 2024-03-29T11:22:48.914 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.933+0000 7f64a07fe640 1 -- 172.21.15.86:0/3726113586 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6498132740 con 0x7f64980b1f60 2024-03-29T11:22:48.914 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.933+0000 7f64a07fe640 1 -- 172.21.15.86:0/3726113586 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6498141090 con 0x7f6498143d10 2024-03-29T11:22:48.914 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.933+0000 7f64a07fe640 1 -- 172.21.15.86:0/3726113586 --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64981421f0 con 0x7f64981542d0 2024-03-29T11:22:48.914 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.933+0000 7f64a07fe640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:48.914 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.933+0000 7f64a07fe640 10 monclient(hunting): authenticate will time out at 993.251844s 2024-03-29T11:22:48.919 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.935+0000 7f648effd640 10 client.?.objecter ms_handle_connect 0x7f6498143d10 2024-03-29T11:22:48.919 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.935+0000 7f648effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:48.919 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.935+0000 7f648effd640 10 client.?.objecter ms_handle_connect 0x7f64981542d0 2024-03-29T11:22:48.919 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.935+0000 7f648effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:48.919 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.935+0000 7f648effd640 1 -- 172.21.15.86:0/3726113586 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4020851802 0 0) 0x7f6498132740 con 0x7f64980b1f60 2024-03-29T11:22:48.919 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.935+0000 7f648effd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:48.919 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.935+0000 7f648effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:48.919 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.935+0000 7f648effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:48.919 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.935+0000 7f648effd640 10 monclient(hunting): my global_id is 4302 2024-03-29T11:22:48.919 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.935+0000 7f648effd640 1 -- 172.21.15.86:0/3726113586 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f647c004eb0 con 0x7f64980b1f60 2024-03-29T11:22:48.919 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.935+0000 7f648effd640 1 -- 172.21.15.86:0/3726113586 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3678545559 0 0) 0x7f6498141090 con 0x7f6498143d10 2024-03-29T11:22:48.920 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.935+0000 7f648effd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:48.920 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.935+0000 7f648effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:48.920 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.935+0000 7f648effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:48.920 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.935+0000 7f648effd640 10 monclient(hunting): my global_id is 4294 2024-03-29T11:22:48.920 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.935+0000 7f648effd640 1 -- 172.21.15.86:0/3726113586 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6498132740 con 0x7f6498143d10 2024-03-29T11:22:48.920 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.935+0000 7f648effd640 1 -- 172.21.15.86:0/3726113586 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1726605222 0 0) 0x7f64981421f0 con 0x7f64981542d0 2024-03-29T11:22:48.920 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.935+0000 7f648effd640 10 monclient(hunting): _negotiate 2024-03-29T11:22:48.920 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.935+0000 7f648effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:48.920 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.935+0000 7f648effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:48.920 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.935+0000 7f648effd640 10 monclient(hunting): my global_id is 4319 2024-03-29T11:22:48.920 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.935+0000 7f648effd640 1 -- 172.21.15.86:0/3726113586 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6498141090 con 0x7f64981542d0 2024-03-29T11:22:48.920 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.936+0000 7f648effd640 1 -- 172.21.15.86:0/3726113586 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2766401368 0 0) 0x7f647c004eb0 con 0x7f64980b1f60 2024-03-29T11:22:48.920 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.936+0000 7f648effd640 1 -- 172.21.15.86:0/3726113586 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f64981421f0 con 0x7f64980b1f60 2024-03-29T11:22:48.920 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.936+0000 7f648effd640 1 -- 172.21.15.86:0/3726113586 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4118502138 0 0) 0x7f6498132740 con 0x7f6498143d10 2024-03-29T11:22:48.921 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.936+0000 7f648effd640 1 -- 172.21.15.86:0/3726113586 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f647c004eb0 con 0x7f6498143d10 2024-03-29T11:22:48.921 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.936+0000 7f648effd640 1 -- 172.21.15.86:0/3726113586 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4018311015 0 0) 0x7f6498141090 con 0x7f64981542d0 2024-03-29T11:22:48.921 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.937+0000 7f648effd640 1 -- 172.21.15.86:0/3726113586 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6498132740 con 0x7f64981542d0 2024-03-29T11:22:48.921 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.937+0000 7f648effd640 1 -- 172.21.15.86:0/3726113586 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f6490001ce0 con 0x7f64980b1f60 2024-03-29T11:22:48.921 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.937+0000 7f648effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:48.921 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.937+0000 7f648effd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:22:48.921 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.937+0000 7f648effd640 10 monclient(hunting): dump: 2024-03-29T11:22:48.921 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:22:48.921 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:48.921 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:48.921 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:48.921 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:48.921 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:48.921 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:22:48.921 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:22:48.921 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:22:48.922 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:48.922 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.937+0000 7f648effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:48.922 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.937+0000 7f648effd640 1 -- 172.21.15.86:0/3726113586 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f6488001710 con 0x7f6498143d10 2024-03-29T11:22:48.922 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.937+0000 7f648effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:48.922 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.937+0000 7f648effd640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:22:48.922 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.937+0000 7f648effd640 10 monclient(hunting): dump: 2024-03-29T11:22:48.922 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:22:48.922 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:48.922 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:48.922 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:48.922 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:48.922 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:48.922 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:22:48.922 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:22:48.923 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:22:48.923 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:48.923 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.937+0000 7f648effd640 1 -- 172.21.15.86:0/3726113586 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f6494001b70 con 0x7f64981542d0 2024-03-29T11:22:48.923 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.937+0000 7f648effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:48.923 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.937+0000 7f648effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:48.923 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.937+0000 7f648effd640 10 monclient(hunting): dump: 2024-03-29T11:22:48.923 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:22:48.923 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:48.923 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:48.923 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:48.923 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:48.923 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:48.923 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:22:48.923 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:22:48.923 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:22:48.923 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:48.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.939+0000 7f648effd640 1 -- 172.21.15.86:0/3726113586 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4270381164 0 0) 0x7f6498132740 con 0x7f64981542d0 2024-03-29T11:22:48.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.939+0000 7f648effd640 1 -- 172.21.15.86:0/3726113586 >> v1:172.21.15.196:6789/0 conn(0x7f6498143d10 legacy=0x7f6498131920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:48.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.939+0000 7f648effd640 1 -- 172.21.15.86:0/3726113586 >> v1:172.21.15.86:6789/0 conn(0x7f64980b1f60 legacy=0x7f64981428f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:48.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.940+0000 7f648effd640 10 monclient: _finish_hunting 0 2024-03-29T11:22:48.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.940+0000 7f648effd640 1 monclient: found mon.c 2024-03-29T11:22:48.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.940+0000 7f648effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:48.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.940+0000 7f648effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:22:48.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.940+0000 7f648effd640 1 -- 172.21.15.86:0/3726113586 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64981424d0 con 0x7f64981542d0 2024-03-29T11:22:48.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.940+0000 7f648effd640 10 monclient: _finish_auth 0 2024-03-29T11:22:48.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.940+0000 7f648effd640 10 monclient: _check_auth_tickets 2024-03-29T11:22:48.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.940+0000 7f648effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:48.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.940+0000 7f64a07fe640 5 monclient: authenticate success, global_id 4319 2024-03-29T11:22:48.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.940+0000 7f64a07fe640 10 monclient: _renew_subs 2024-03-29T11:22:48.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.940+0000 7f64a07fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:22:48.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.940+0000 7f64a07fe640 1 -- 172.21.15.86:0/3726113586 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6498143000 con 0x7f64981542d0 2024-03-29T11:22:48.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.940+0000 7f64a07fe640 10 client.4319.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:48.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.940+0000 7f64a07fe640 10 monclient: _renew_subs 2024-03-29T11:22:48.925 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.940+0000 7f64a07fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:22:48.925 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.940+0000 7f64a07fe640 1 -- 172.21.15.86:0/3726113586 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6498155050 con 0x7f64981542d0 2024-03-29T11:22:48.925 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.940+0000 7f64a07fe640 1 librados: init done 2024-03-29T11:22:48.925 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.941+0000 7f648effd640 1 -- 172.21.15.86:0/3726113586 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6494000fe0 con 0x7f64981542d0 2024-03-29T11:22:48.925 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.941+0000 7f648effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:48.926 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.941+0000 7f648ffff640 4 set_mon_vals no callback set 2024-03-29T11:22:48.926 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.942+0000 7f648effd640 1 -- 172.21.15.86:0/3726113586 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f6494004a90 con 0x7f64981542d0 2024-03-29T11:22:48.927 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.942+0000 7f648effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:48.927 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.942+0000 7f648effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:48.927 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.942+0000 7f648effd640 10 monclient: dump: 2024-03-29T11:22:48.927 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:22:48.927 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:48.927 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:48.927 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:48.927 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:48.927 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:48.927 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:22:48.927 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:22:48.927 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:22:48.927 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:48.927 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.942+0000 7f648effd640 1 -- 172.21.15.86:0/3726113586 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f649401aa60 con 0x7f64981542d0 2024-03-29T11:22:48.927 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.943+0000 7f649e59a640 10 monclient: get_auth_request con 0x7f647c06e2c0 auth_method 0 2024-03-29T11:22:48.927 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.943+0000 7f648effd640 1 -- 172.21.15.86:0/3726113586 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f64940847f0 con 0x7f64981542d0 2024-03-29T11:22:48.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.943+0000 7f648effd640 10 client.4319.objecter ms_dispatch 0x7f6498000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:48.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.943+0000 7f648effd640 3 client.4319.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:48.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.943+0000 7f648effd640 3 client.4319.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:48.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.944+0000 7f648effd640 20 client.4319.objecter dump_active .. 0 homeless 2024-03-29T11:22:48.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.944+0000 7f648effd640 10 client.4319.objecter ms_handle_connect 0x7f647c06e2c0 2024-03-29T11:22:48.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.944+0000 7f64a07fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:48.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.944+0000 7f64a07fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:48.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.944+0000 7f64a07fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:22:48.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.944+0000 7f64a07fe640 1 -- 172.21.15.86:0/3726113586 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6480005080 con 0x7f64981542d0 2024-03-29T11:22:48.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.955+0000 7f648effd640 1 -- 172.21.15.86:0/3726113586 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6494050150 con 0x7f64981542d0 2024-03-29T11:22:48.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.955+0000 7f648effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:48.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:48.955+0000 7f648effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:49.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:49.341+0000 7f64a07fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:49.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:49.341+0000 7f64a07fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:49.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:49.341+0000 7f64a07fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:22:49.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:49.341+0000 7f64a07fe640 1 -- 172.21.15.86:0/3726113586 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6480005330 con 0x7f64981542d0 2024-03-29T11:22:49.319 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:49.342+0000 7f648effd640 1 -- 172.21.15.86:0/3726113586 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f6494052080 con 0x7f64981542d0 2024-03-29T11:22:49.319 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:49.342+0000 7f648effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:49.319 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:49.342+0000 7f648effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:49.319 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:22:49.320 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:49.327 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:49.346+0000 7f64a07fe640 10 librados: watch_flush enter 2024-03-29T11:22:49.327 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:49.346+0000 7f64a07fe640 10 librados: watch_flush exit 2024-03-29T11:22:49.327 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:49.346+0000 7f64a07fe640 20 client.4319.objecter shutdown clearing up homeless session... 2024-03-29T11:22:49.327 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:49.346+0000 7f64a07fe640 10 client.4319.objecter successfully canceled tick 2024-03-29T11:22:49.327 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:49.346+0000 7f64a07fe640 1 -- 172.21.15.86:0/3726113586 >> v1:172.21.15.196:6816/588110318 conn(0x7f647c06e2c0 legacy=0x7f647c070780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:49.327 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:49.346+0000 7f64a07fe640 10 monclient: shutdown 2024-03-29T11:22:49.327 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:49.346+0000 7f64a07fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:49.327 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:49.346+0000 7f64a07fe640 1 -- 172.21.15.86:0/3726113586 >> v1:172.21.15.86:6790/0 conn(0x7f64981542d0 legacy=0x7f6498132030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:49.327 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:49.347+0000 7f64a07fe640 1 -- 172.21.15.86:0/3726113586 shutdown_connections 2024-03-29T11:22:49.328 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:49.347+0000 7f64a07fe640 1 -- 172.21.15.86:0/3726113586 >> 172.21.15.86:0/3726113586 conn(0x7f6498066a00 msgr2=0x7f64980b7bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:49.328 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:49.347+0000 7f64a07fe640 1 -- 172.21.15.86:0/3726113586 shutdown_connections 2024-03-29T11:22:49.328 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:49.347+0000 7f64a07fe640 1 -- 172.21.15.86:0/3726113586 wait complete. 2024-03-29T11:22:49.328 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:49.347+0000 7f64a07fe640 1 librados: shutdown 2024-03-29T11:22:49.341 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:22:56.147 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:22:56.291 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.313+0000 7f0d8167b640 10 monclient: get_monmap_and_config 2024-03-29T11:22:56.291 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.313+0000 7f0d8167b640 10 monclient: build_initial_monmap 2024-03-29T11:22:56.291 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.313+0000 7f0d8167b640 10 monclient: monmap: 2024-03-29T11:22:56.291 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:22:56.291 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:56.291 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:22:56.314454+0000 2024-03-29T11:22:56.291 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:22:56.314454+0000 2024-03-29T11:22:56.292 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:56.292 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:56.292 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:22:56.292 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:22:56.292 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:22:56.292 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:56.294 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.314+0000 7f0d8167b640 1 Processor -- start 2024-03-29T11:22:56.294 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.314+0000 7f0d8167b640 1 -- start start 2024-03-29T11:22:56.294 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.314+0000 7f0d8167b640 10 monclient: init 2024-03-29T11:22:56.294 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.315+0000 7f0d8167b640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:56.294 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.315+0000 7f0d8167b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:22:56.294 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.315+0000 7f0d8167b640 10 monclient(hunting): picked mon.noname-a con 0x7f0d7c0b0860 addr v1:172.21.15.86:6789/0 2024-03-29T11:22:56.294 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.316+0000 7f0d8167b640 10 monclient(hunting): picked mon.noname-b con 0x7f0d7c13ef30 addr v1:172.21.15.86:6790/0 2024-03-29T11:22:56.294 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.316+0000 7f0d8167b640 10 monclient(hunting): picked mon.noname-c con 0x7f0d7c0af8e0 addr v1:172.21.15.196:6789/0 2024-03-29T11:22:56.294 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.316+0000 7f0d8167b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d7c0b0c40 con 0x7f0d7c0b0860 2024-03-29T11:22:56.294 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.316+0000 7f0d7affd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f0d7c0b0860 0x7f0d7c13e820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:51692/0 (socket says 172.21.15.86:51692) 2024-03-29T11:22:56.294 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.316+0000 7f0d7affd640 1 -- 172.21.15.86:0/638307550 learned_addr learned my addr 172.21.15.86:0/638307550 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:22:56.295 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.316+0000 7f0d8167b640 1 -- 172.21.15.86:0/638307550 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d7c0aed10 con 0x7f0d7c0af8e0 2024-03-29T11:22:56.295 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.316+0000 7f0d8167b640 1 -- 172.21.15.86:0/638307550 --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d7c0afd50 con 0x7f0d7c13ef30 2024-03-29T11:22:56.295 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.316+0000 7f0d8167b640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:56.295 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.316+0000 7f0d8167b640 10 monclient(hunting): authenticate will time out at 1000.634847s 2024-03-29T11:22:56.299 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.317+0000 7f0d79ffb640 1 -- 172.21.15.86:0/638307550 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1847988512 0 0) 0x7f0d7c0b0c40 con 0x7f0d7c0b0860 2024-03-29T11:22:56.299 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.317+0000 7f0d79ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:56.299 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.317+0000 7f0d79ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:56.299 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.317+0000 7f0d79ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:56.299 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.317+0000 7f0d79ffb640 10 monclient(hunting): my global_id is 4305 2024-03-29T11:22:56.299 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.317+0000 7f0d79ffb640 1 -- 172.21.15.86:0/638307550 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d64003440 con 0x7f0d7c0b0860 2024-03-29T11:22:56.299 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.317+0000 7f0d79ffb640 1 -- 172.21.15.86:0/638307550 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3168223807 0 0) 0x7f0d7c0afd50 con 0x7f0d7c13ef30 2024-03-29T11:22:56.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.317+0000 7f0d79ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:22:56.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.317+0000 7f0d79ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:56.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.317+0000 7f0d79ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:56.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.317+0000 7f0d79ffb640 10 monclient(hunting): my global_id is 4322 2024-03-29T11:22:56.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.317+0000 7f0d79ffb640 1 -- 172.21.15.86:0/638307550 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d7c0b0c40 con 0x7f0d7c13ef30 2024-03-29T11:22:56.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.318+0000 7f0d79ffb640 1 -- 172.21.15.86:0/638307550 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3769557857 0 0) 0x7f0d7c0b0c40 con 0x7f0d7c13ef30 2024-03-29T11:22:56.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.318+0000 7f0d79ffb640 1 -- 172.21.15.86:0/638307550 >> v1:172.21.15.196:6789/0 conn(0x7f0d7c0af8e0 legacy=0x7f0d7c149a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:56.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.318+0000 7f0d79ffb640 1 -- 172.21.15.86:0/638307550 >> v1:172.21.15.86:6789/0 conn(0x7f0d7c0b0860 legacy=0x7f0d7c13e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:56.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.318+0000 7f0d79ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:22:56.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.318+0000 7f0d79ffb640 1 monclient: found mon.noname-b 2024-03-29T11:22:56.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.318+0000 7f0d79ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:56.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.318+0000 7f0d79ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:22:56.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.318+0000 7f0d79ffb640 1 -- 172.21.15.86:0/638307550 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d7c0ab1b0 con 0x7f0d7c13ef30 2024-03-29T11:22:56.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.318+0000 7f0d79ffb640 10 monclient: _finish_auth 0 2024-03-29T11:22:56.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.318+0000 7f0d79ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:22:56.301 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.318+0000 7f0d79ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:56.301 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.318+0000 7f0d79ffb640 1 -- 172.21.15.86:0/638307550 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f0d70001670 con 0x7f0d7c13ef30 2024-03-29T11:22:56.301 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.318+0000 7f0d79ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:56.301 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.318+0000 7f0d79ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:22:56.301 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.318+0000 7f0d79ffb640 10 monclient: dump: 2024-03-29T11:22:56.301 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:22:56.301 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:56.301 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:56.301 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:56.301 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:56.301 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:56.301 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:22:56.301 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:22:56.301 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:22:56.301 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:56.301 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.318+0000 7f0d8167b640 5 monclient: authenticate success, global_id 4322 2024-03-29T11:22:56.302 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.318+0000 7f0d8167b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:22:56.302 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.318+0000 7f0d79ffb640 1 -- 172.21.15.86:0/638307550 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d700017d0 con 0x7f0d7c13ef30 2024-03-29T11:22:56.302 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.318+0000 7f0d79ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:56.302 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.318+0000 7f0d79ffb640 1 -- 172.21.15.86:0/638307550 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f0d70001ba0 con 0x7f0d7c13ef30 2024-03-29T11:22:56.302 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.318+0000 7f0d8167b640 10 monclient: get_monmap_and_config success 2024-03-29T11:22:56.302 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.318+0000 7f0d8167b640 4 set_mon_vals no callback set 2024-03-29T11:22:56.302 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.318+0000 7f0d79ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:56.302 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.318+0000 7f0d79ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:56.302 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.318+0000 7f0d79ffb640 10 monclient: dump: 2024-03-29T11:22:56.302 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:22:56.302 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:56.302 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:56.302 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:56.302 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:56.303 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:56.303 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:22:56.303 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:22:56.303 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:22:56.303 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:56.303 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.319+0000 7f0d8167b640 10 monclient: shutdown 2024-03-29T11:22:56.303 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.319+0000 7f0d8167b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:56.303 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.319+0000 7f0d8167b640 1 -- 172.21.15.86:0/638307550 >> v1:172.21.15.86:6790/0 conn(0x7f0d7c13ef30 legacy=0x7f0d7c0af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:56.303 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.319+0000 7f0d8167b640 1 -- 172.21.15.86:0/638307550 shutdown_connections 2024-03-29T11:22:56.303 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.319+0000 7f0d8167b640 1 -- 172.21.15.86:0/638307550 >> 172.21.15.86:0/638307550 conn(0x7f0d7c06b660 msgr2=0x7f0d7c06ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:56.303 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.320+0000 7f0d8167b640 1 -- 172.21.15.86:0/638307550 shutdown_connections 2024-03-29T11:22:56.303 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.320+0000 7f0d8167b640 1 -- 172.21.15.86:0/638307550 wait complete. 2024-03-29T11:22:56.303 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.321+0000 7f0d8167b640 10 monclient: build_initial_monmap 2024-03-29T11:22:56.303 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.321+0000 7f0d8167b640 10 monclient: monmap: 2024-03-29T11:22:56.303 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:22:56.303 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:22:56.304 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:22:56.304 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:22:56.304 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:22:56.304 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:56.304 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:22:56.304 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:22:56.304 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:22:56.304 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:56.304 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.321+0000 7f0d8167b640 1 librados: starting msgr at 2024-03-29T11:22:56.304 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.321+0000 7f0d8167b640 1 librados: starting objecter 2024-03-29T11:22:56.304 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.321+0000 7f0d8167b640 1 Processor -- start 2024-03-29T11:22:56.304 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.322+0000 7f0d8167b640 1 -- start start 2024-03-29T11:22:56.304 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.322+0000 7f0d8167b640 1 librados: setting wanted keys 2024-03-29T11:22:56.311 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.322+0000 7f0d8167b640 1 librados: calling monclient init 2024-03-29T11:22:56.311 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.322+0000 7f0d8167b640 10 monclient: init 2024-03-29T11:22:56.311 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.323+0000 7f0d8167b640 10 monclient: _reopen_session rank -1 2024-03-29T11:22:56.311 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.323+0000 7f0d8167b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:22:56.312 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.323+0000 7f0d8167b640 10 monclient(hunting): picked mon.noname-a con 0x7f0d7c0af8e0 addr v1:172.21.15.86:6789/0 2024-03-29T11:22:56.312 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.323+0000 7f0d7affd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f0d7c0af8e0 0x7f0d7c1c8080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:52004/0 (socket says 172.21.15.86:52004) 2024-03-29T11:22:56.312 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.323+0000 7f0d7affd640 1 -- 172.21.15.86:0/3100194055 learned_addr learned my addr 172.21.15.86:0/3100194055 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:22:56.312 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.323+0000 7f0d8167b640 10 monclient(hunting): picked mon.noname-b con 0x7f0d7c0b0860 addr v1:172.21.15.196:6789/0 2024-03-29T11:22:56.312 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.323+0000 7f0d637fe640 10 client.?.objecter ms_handle_connect 0x7f0d7c0af8e0 2024-03-29T11:22:56.312 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.323+0000 7f0d637fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:56.312 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.323+0000 7f0d8167b640 10 monclient(hunting): picked mon.noname-c con 0x7f0d7c13ef30 addr v1:172.21.15.86:6790/0 2024-03-29T11:22:56.312 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.323+0000 7f0d8167b640 1 -- 172.21.15.86:0/3100194055 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d7c13a460 con 0x7f0d7c0af8e0 2024-03-29T11:22:56.312 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.323+0000 7f0d8167b640 1 -- 172.21.15.86:0/3100194055 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d7c13b430 con 0x7f0d7c0b0860 2024-03-29T11:22:56.312 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.323+0000 7f0d8167b640 1 -- 172.21.15.86:0/3100194055 --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d7c13b710 con 0x7f0d7c13ef30 2024-03-29T11:22:56.312 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.323+0000 7f0d8167b640 10 monclient(hunting): _renew_subs 2024-03-29T11:22:56.312 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.323+0000 7f0d8167b640 10 monclient(hunting): authenticate will time out at 1000.641970s 2024-03-29T11:22:56.312 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.324+0000 7f0d637fe640 10 client.?.objecter ms_handle_connect 0x7f0d7c13ef30 2024-03-29T11:22:56.312 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.324+0000 7f0d637fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:56.312 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.324+0000 7f0d637fe640 1 -- 172.21.15.86:0/3100194055 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3903557980 0 0) 0x7f0d7c13b710 con 0x7f0d7c13ef30 2024-03-29T11:22:56.313 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.324+0000 7f0d637fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:56.313 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.324+0000 7f0d637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:56.313 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.324+0000 7f0d637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:56.313 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.324+0000 7f0d637fe640 10 monclient(hunting): my global_id is 4325 2024-03-29T11:22:56.313 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.324+0000 7f0d637fe640 1 -- 172.21.15.86:0/3100194055 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d68004eb0 con 0x7f0d7c13ef30 2024-03-29T11:22:56.313 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.324+0000 7f0d637fe640 1 -- 172.21.15.86:0/3100194055 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 81234392 0 0) 0x7f0d7c13a460 con 0x7f0d7c0af8e0 2024-03-29T11:22:56.313 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.324+0000 7f0d637fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:56.313 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.324+0000 7f0d637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:56.313 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.324+0000 7f0d637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:56.313 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.324+0000 7f0d637fe640 10 monclient(hunting): my global_id is 4308 2024-03-29T11:22:56.313 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.324+0000 7f0d637fe640 1 -- 172.21.15.86:0/3100194055 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d7c13b710 con 0x7f0d7c0af8e0 2024-03-29T11:22:56.313 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.324+0000 7f0d637fe640 1 -- 172.21.15.86:0/3100194055 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2367419773 0 0) 0x7f0d68004eb0 con 0x7f0d7c13ef30 2024-03-29T11:22:56.313 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.324+0000 7f0d637fe640 1 -- 172.21.15.86:0/3100194055 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f0d7c13a460 con 0x7f0d7c13ef30 2024-03-29T11:22:56.313 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.324+0000 7f0d637fe640 1 -- 172.21.15.86:0/3100194055 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f0d74001b60 con 0x7f0d7c13ef30 2024-03-29T11:22:56.313 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.324+0000 7f0d637fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:56.314 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.324+0000 7f0d637fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:22:56.314 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.324+0000 7f0d637fe640 10 monclient(hunting): dump: 2024-03-29T11:22:56.314 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:22:56.314 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:56.314 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:56.314 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:56.314 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:56.314 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:56.314 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:22:56.314 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:22:56.314 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:22:56.314 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:56.314 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.324+0000 7f0d637fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:22:56.314 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.325+0000 7f0d637fe640 10 client.?.objecter ms_handle_connect 0x7f0d7c0b0860 2024-03-29T11:22:56.314 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.325+0000 7f0d637fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:22:56.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.325+0000 7f0d637fe640 1 -- 172.21.15.86:0/3100194055 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2737753130 0 0) 0x7f0d7c13b430 con 0x7f0d7c0b0860 2024-03-29T11:22:56.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.325+0000 7f0d637fe640 10 monclient(hunting): _negotiate 2024-03-29T11:22:56.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.325+0000 7f0d637fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:22:56.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.325+0000 7f0d637fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:22:56.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.325+0000 7f0d637fe640 10 monclient(hunting): my global_id is 4300 2024-03-29T11:22:56.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.325+0000 7f0d637fe640 1 -- 172.21.15.86:0/3100194055 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d68004eb0 con 0x7f0d7c0b0860 2024-03-29T11:22:56.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.325+0000 7f0d637fe640 1 -- 172.21.15.86:0/3100194055 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 708068266 0 0) 0x7f0d7c13a460 con 0x7f0d7c13ef30 2024-03-29T11:22:56.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.325+0000 7f0d637fe640 1 -- 172.21.15.86:0/3100194055 >> v1:172.21.15.196:6789/0 conn(0x7f0d7c0b0860 legacy=0x7f0d7c1dee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:56.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.325+0000 7f0d637fe640 1 -- 172.21.15.86:0/3100194055 >> v1:172.21.15.86:6789/0 conn(0x7f0d7c0af8e0 legacy=0x7f0d7c1c8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:56.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.325+0000 7f0d637fe640 10 monclient: _finish_hunting 0 2024-03-29T11:22:56.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.325+0000 7f0d637fe640 1 monclient: found mon.c 2024-03-29T11:22:56.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.325+0000 7f0d637fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:22:56.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.325+0000 7f0d637fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:22:56.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.325+0000 7f0d637fe640 1 -- 172.21.15.86:0/3100194055 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d7c1390b0 con 0x7f0d7c13ef30 2024-03-29T11:22:56.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.325+0000 7f0d637fe640 10 monclient: _finish_auth 0 2024-03-29T11:22:56.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.325+0000 7f0d637fe640 10 monclient: _check_auth_tickets 2024-03-29T11:22:56.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.325+0000 7f0d637fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:22:56.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.325+0000 7f0d8167b640 5 monclient: authenticate success, global_id 4325 2024-03-29T11:22:56.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.325+0000 7f0d8167b640 10 monclient: _renew_subs 2024-03-29T11:22:56.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.325+0000 7f0d8167b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:22:56.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.325+0000 7f0d8167b640 1 -- 172.21.15.86:0/3100194055 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d7c139bc0 con 0x7f0d7c13ef30 2024-03-29T11:22:56.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.325+0000 7f0d8167b640 10 client.4325.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:22:56.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.325+0000 7f0d8167b640 10 monclient: _renew_subs 2024-03-29T11:22:56.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.325+0000 7f0d8167b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:22:56.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.326+0000 7f0d8167b640 1 -- 172.21.15.86:0/3100194055 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d7c13bdd0 con 0x7f0d7c13ef30 2024-03-29T11:22:56.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.326+0000 7f0d8167b640 1 librados: init done 2024-03-29T11:22:56.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.326+0000 7f0d637fe640 1 -- 172.21.15.86:0/3100194055 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f0d74002050 con 0x7f0d7c13ef30 2024-03-29T11:22:56.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.326+0000 7f0d637fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:22:56.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.326+0000 7f0d797fa640 4 set_mon_vals no callback set 2024-03-29T11:22:56.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.326+0000 7f0d637fe640 1 -- 172.21.15.86:0/3100194055 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f0d74003b30 con 0x7f0d7c13ef30 2024-03-29T11:22:56.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.327+0000 7f0d637fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:22:56.317 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.327+0000 7f0d637fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:22:56.317 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.327+0000 7f0d637fe640 10 monclient: dump: 2024-03-29T11:22:56.317 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:22:56.317 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:22:56.317 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:56.317 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:22:56.317 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:22:56.317 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:22:56.317 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:22:56.317 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:22:56.317 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:22:56.317 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:22:56.317 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.327+0000 7f0d8167b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:56.317 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.328+0000 7f0d8167b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:56.317 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.328+0000 7f0d8167b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:22:56.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.328+0000 7f0d8167b640 1 -- 172.21.15.86:0/3100194055 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d5c005080 con 0x7f0d7c13ef30 2024-03-29T11:22:56.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.328+0000 7f0d637fe640 1 -- 172.21.15.86:0/3100194055 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f0d74019b00 con 0x7f0d7c13ef30 2024-03-29T11:22:56.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.329+0000 7f0d7affd640 10 monclient: get_auth_request con 0x7f0d6806dc60 auth_method 0 2024-03-29T11:22:56.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.329+0000 7f0d637fe640 1 -- 172.21.15.86:0/3100194055 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f0d74083ad0 con 0x7f0d7c13ef30 2024-03-29T11:22:56.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.329+0000 7f0d637fe640 10 client.4325.objecter ms_dispatch 0x7f0d7c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:22:56.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.329+0000 7f0d637fe640 3 client.4325.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:22:56.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.329+0000 7f0d637fe640 3 client.4325.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:22:56.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.329+0000 7f0d637fe640 20 client.4325.objecter dump_active .. 0 homeless 2024-03-29T11:22:56.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.329+0000 7f0d637fe640 10 client.4325.objecter ms_handle_connect 0x7f0d6806dc60 2024-03-29T11:22:56.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.335+0000 7f0d637fe640 1 -- 172.21.15.86:0/3100194055 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f0d74085e60 con 0x7f0d7c13ef30 2024-03-29T11:22:56.318 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.336+0000 7f0d637fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:22:56.319 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.336+0000 7f0d637fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:22:56.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.710+0000 7f0d8167b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:56.689 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.710+0000 7f0d8167b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:56.689 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.710+0000 7f0d8167b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:22:56.689 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.710+0000 7f0d8167b640 1 -- 172.21.15.86:0/3100194055 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0d5c005330 con 0x7f0d7c13ef30 2024-03-29T11:22:56.689 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.711+0000 7f0d637fe640 1 -- 172.21.15.86:0/3100194055 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f0d7404f3b0 con 0x7f0d7c13ef30 2024-03-29T11:22:56.689 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.711+0000 7f0d637fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:22:56.690 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.711+0000 7f0d637fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:22:56.690 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:22:56.690 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:22:56.693 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.715+0000 7f0d8167b640 10 librados: watch_flush enter 2024-03-29T11:22:56.693 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.715+0000 7f0d8167b640 10 librados: watch_flush exit 2024-03-29T11:22:56.693 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.715+0000 7f0d8167b640 20 client.4325.objecter shutdown clearing up homeless session... 2024-03-29T11:22:56.693 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.715+0000 7f0d8167b640 10 client.4325.objecter successfully canceled tick 2024-03-29T11:22:56.693 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.715+0000 7f0d8167b640 1 -- 172.21.15.86:0/3100194055 >> v1:172.21.15.196:6816/588110318 conn(0x7f0d6806dc60 legacy=0x7f0d68070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:56.693 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.715+0000 7f0d8167b640 10 monclient: shutdown 2024-03-29T11:22:56.693 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.715+0000 7f0d8167b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:22:56.693 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.715+0000 7f0d8167b640 1 -- 172.21.15.86:0/3100194055 >> v1:172.21.15.86:6790/0 conn(0x7f0d7c13ef30 legacy=0x7f0d7c1e60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:22:56.693 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.715+0000 7f0d8167b640 1 -- 172.21.15.86:0/3100194055 shutdown_connections 2024-03-29T11:22:56.693 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.715+0000 7f0d8167b640 1 -- 172.21.15.86:0/3100194055 >> 172.21.15.86:0/3100194055 conn(0x7f0d7c06b660 msgr2=0x7f0d7c13e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:22:56.694 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.716+0000 7f0d8167b640 1 -- 172.21.15.86:0/3100194055 shutdown_connections 2024-03-29T11:22:56.694 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.716+0000 7f0d8167b640 1 -- 172.21.15.86:0/3100194055 wait complete. 2024-03-29T11:22:56.694 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:22:56.716+0000 7f0d8167b640 1 librados: shutdown 2024-03-29T11:22:56.707 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:03.512 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:03.657 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.679+0000 7f2988d15640 10 monclient: get_monmap_and_config 2024-03-29T11:23:03.657 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.679+0000 7f2988d15640 10 monclient: build_initial_monmap 2024-03-29T11:23:03.657 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.679+0000 7f2988d15640 10 monclient: monmap: 2024-03-29T11:23:03.657 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:23:03.657 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:03.658 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:23:03.680583+0000 2024-03-29T11:23:03.658 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:23:03.680583+0000 2024-03-29T11:23:03.658 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:03.658 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:03.658 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:23:03.658 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:23:03.658 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:23:03.658 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:03.659 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.680+0000 7f2988d15640 1 Processor -- start 2024-03-29T11:23:03.659 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.680+0000 7f2988d15640 1 -- start start 2024-03-29T11:23:03.659 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.680+0000 7f2988d15640 10 monclient: init 2024-03-29T11:23:03.659 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.681+0000 7f2988d15640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:03.661 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.681+0000 7f2988d15640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:23:03.661 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.681+0000 7f2988d15640 10 monclient(hunting): picked mon.noname-a con 0x7f29841457f0 addr v1:172.21.15.86:6789/0 2024-03-29T11:23:03.661 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.681+0000 7f2988d15640 10 monclient(hunting): picked mon.noname-c con 0x7f298413f650 addr v1:172.21.15.196:6789/0 2024-03-29T11:23:03.661 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.681+0000 7f2988d15640 10 monclient(hunting): picked mon.noname-b con 0x7f29840a9d70 addr v1:172.21.15.86:6790/0 2024-03-29T11:23:03.661 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.681+0000 7f2988d15640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29840aaef0 con 0x7f29841457f0 2024-03-29T11:23:03.662 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.681+0000 7f2988d15640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2984145bd0 con 0x7f298413f650 2024-03-29T11:23:03.662 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.681+0000 7f2988d15640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2984145eb0 con 0x7f29840a9d70 2024-03-29T11:23:03.662 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.681+0000 7f2988d15640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:03.662 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.681+0000 7f2988d15640 10 monclient(hunting): authenticate will time out at 1007.999650s 2024-03-29T11:23:03.662 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.683+0000 7f298259c640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f29841457f0 0x7f298413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:47090/0 (socket says 172.21.15.86:47090) 2024-03-29T11:23:03.662 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.683+0000 7f298259c640 1 -- 172.21.15.86:0/2726566737 learned_addr learned my addr 172.21.15.86:0/2726566737 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:23:03.663 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.684+0000 7f298159a640 1 -- 172.21.15.86:0/2726566737 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 26516263 0 0) 0x7f29840aaef0 con 0x7f29841457f0 2024-03-29T11:23:03.663 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.684+0000 7f298159a640 10 monclient(hunting): _negotiate 2024-03-29T11:23:03.663 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.684+0000 7f298159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:03.663 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.684+0000 7f298159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:03.663 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.684+0000 7f298159a640 10 monclient(hunting): my global_id is 4311 2024-03-29T11:23:03.663 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.684+0000 7f298159a640 1 -- 172.21.15.86:0/2726566737 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f296c003520 con 0x7f29841457f0 2024-03-29T11:23:03.664 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.685+0000 7f298159a640 1 -- 172.21.15.86:0/2726566737 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 41803668 0 0) 0x7f2984145eb0 con 0x7f29840a9d70 2024-03-29T11:23:03.664 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.685+0000 7f298159a640 10 monclient(hunting): _negotiate 2024-03-29T11:23:03.664 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.685+0000 7f298159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:03.664 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.685+0000 7f298159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:03.664 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.685+0000 7f298159a640 10 monclient(hunting): my global_id is 4328 2024-03-29T11:23:03.664 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.685+0000 7f298159a640 1 -- 172.21.15.86:0/2726566737 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29840aaef0 con 0x7f29840a9d70 2024-03-29T11:23:03.664 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.685+0000 7f298159a640 1 -- 172.21.15.86:0/2726566737 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 461317718 0 0) 0x7f296c003520 con 0x7f29841457f0 2024-03-29T11:23:03.665 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.685+0000 7f298159a640 1 -- 172.21.15.86:0/2726566737 >> v1:172.21.15.86:6790/0 conn(0x7f29840a9d70 legacy=0x7f298414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:03.665 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.685+0000 7f298159a640 1 -- 172.21.15.86:0/2726566737 >> v1:172.21.15.196:6789/0 conn(0x7f298413f650 legacy=0x7f29840b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:03.666 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.685+0000 7f298159a640 10 monclient: _finish_hunting 0 2024-03-29T11:23:03.666 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.685+0000 7f298159a640 1 monclient: found mon.noname-a 2024-03-29T11:23:03.666 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.685+0000 7f298159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:03.666 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.685+0000 7f298159a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.86:6789/0 2024-03-29T11:23:03.666 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.685+0000 7f298159a640 1 -- 172.21.15.86:0/2726566737 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2984146340 con 0x7f29841457f0 2024-03-29T11:23:03.666 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.685+0000 7f298159a640 10 monclient: _finish_auth 0 2024-03-29T11:23:03.666 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.685+0000 7f298159a640 10 monclient: _check_auth_tickets 2024-03-29T11:23:03.666 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.685+0000 7f298159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:03.666 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.685+0000 7f2988d15640 5 monclient: authenticate success, global_id 4311 2024-03-29T11:23:03.666 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.685+0000 7f2988d15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:03.666 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.685+0000 7f298159a640 1 -- 172.21.15.86:0/2726566737 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f2974001610 con 0x7f29841457f0 2024-03-29T11:23:03.667 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.685+0000 7f298159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:03.667 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.685+0000 7f298159a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:03.667 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.685+0000 7f298159a640 10 monclient: dump: 2024-03-29T11:23:03.667 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:03.667 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:03.667 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:03.667 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:03.667 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:03.667 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:03.667 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:03.667 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:03.667 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:03.667 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:03.667 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.685+0000 7f2988d15640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:03.667 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.688+0000 7f298159a640 1 -- 172.21.15.86:0/2726566737 <== mon.0 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2974001830 con 0x7f29841457f0 2024-03-29T11:23:03.667 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.688+0000 7f298159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:03.668 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.688+0000 7f2988d15640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:03.668 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.688+0000 7f2988d15640 4 set_mon_vals no callback set 2024-03-29T11:23:03.668 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.689+0000 7f298159a640 1 -- 172.21.15.86:0/2726566737 <== mon.0 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f2974001bc0 con 0x7f29841457f0 2024-03-29T11:23:03.668 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.689+0000 7f298159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:03.668 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.689+0000 7f298159a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:03.668 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.689+0000 7f298159a640 10 monclient: dump: 2024-03-29T11:23:03.668 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:03.668 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:03.668 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:03.668 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:03.669 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:03.669 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:03.669 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:03.669 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:03.669 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:03.669 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:03.670 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.691+0000 7f2988d15640 10 monclient: shutdown 2024-03-29T11:23:03.670 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.691+0000 7f2988d15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:03.670 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.691+0000 7f2988d15640 1 -- 172.21.15.86:0/2726566737 >> v1:172.21.15.86:6789/0 conn(0x7f29841457f0 legacy=0x7f298413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:03.670 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.691+0000 7f2988d15640 1 -- 172.21.15.86:0/2726566737 shutdown_connections 2024-03-29T11:23:03.670 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.691+0000 7f2988d15640 1 -- 172.21.15.86:0/2726566737 >> 172.21.15.86:0/2726566737 conn(0x7f2984066a00 msgr2=0x7f2984066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:03.670 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.692+0000 7f2988d15640 1 -- 172.21.15.86:0/2726566737 shutdown_connections 2024-03-29T11:23:03.670 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.692+0000 7f2988d15640 1 -- 172.21.15.86:0/2726566737 wait complete. 2024-03-29T11:23:03.671 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.694+0000 7f2988d15640 10 monclient: build_initial_monmap 2024-03-29T11:23:03.671 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.694+0000 7f2988d15640 10 monclient: monmap: 2024-03-29T11:23:03.672 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:23:03.672 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:03.672 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:23:03.672 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:23:03.672 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:03.672 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:03.672 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:23:03.672 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:23:03.672 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:23:03.672 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:03.672 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.694+0000 7f2988d15640 1 librados: starting msgr at 2024-03-29T11:23:03.673 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.694+0000 7f2988d15640 1 librados: starting objecter 2024-03-29T11:23:03.673 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.694+0000 7f2988d15640 1 Processor -- start 2024-03-29T11:23:03.673 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.695+0000 7f2988d15640 1 -- start start 2024-03-29T11:23:03.674 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.695+0000 7f2988d15640 1 librados: setting wanted keys 2024-03-29T11:23:03.674 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.695+0000 7f2988d15640 1 librados: calling monclient init 2024-03-29T11:23:03.674 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.695+0000 7f2988d15640 10 monclient: init 2024-03-29T11:23:03.675 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.696+0000 7f2988d15640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:03.675 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.696+0000 7f2988d15640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:03.675 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.696+0000 7f2988d15640 10 monclient(hunting): picked mon.noname-b con 0x7f29840a9d70 addr v1:172.21.15.196:6789/0 2024-03-29T11:23:03.675 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.696+0000 7f2988d15640 10 monclient(hunting): picked mon.noname-a con 0x7f298413f650 addr v1:172.21.15.86:6789/0 2024-03-29T11:23:03.675 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.697+0000 7f2981d9b640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f298413f650 0x7f298406fda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:47740/0 (socket says 172.21.15.86:47740) 2024-03-29T11:23:03.675 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.697+0000 7f2981d9b640 1 -- 172.21.15.86:0/3190058790 learned_addr learned my addr 172.21.15.86:0/3190058790 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:23:03.676 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.698+0000 7f2988d15640 10 monclient(hunting): picked mon.noname-c con 0x7f29841457f0 addr v1:172.21.15.86:6790/0 2024-03-29T11:23:03.676 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.698+0000 7f2988d15640 1 -- 172.21.15.86:0/3190058790 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29841dfae0 con 0x7f298413f650 2024-03-29T11:23:03.676 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.698+0000 7f2988d15640 1 -- 172.21.15.86:0/3190058790 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2984071110 con 0x7f29840a9d70 2024-03-29T11:23:03.676 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.698+0000 7f2988d15640 1 -- 172.21.15.86:0/3190058790 --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29841dbee0 con 0x7f29841457f0 2024-03-29T11:23:03.676 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.698+0000 7f2988d15640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:03.676 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.698+0000 7f2988d15640 10 monclient(hunting): authenticate will time out at 1008.016060s 2024-03-29T11:23:03.677 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.699+0000 7f2972ffd640 10 client.?.objecter ms_handle_connect 0x7f298413f650 2024-03-29T11:23:03.677 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.699+0000 7f2972ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:03.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.702+0000 7f2972ffd640 10 client.?.objecter ms_handle_connect 0x7f29841457f0 2024-03-29T11:23:03.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.702+0000 7f2972ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:03.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.702+0000 7f2972ffd640 10 client.?.objecter ms_handle_connect 0x7f29840a9d70 2024-03-29T11:23:03.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.702+0000 7f2972ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:03.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.702+0000 7f2972ffd640 1 -- 172.21.15.86:0/3190058790 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2414070880 0 0) 0x7f29841dfae0 con 0x7f298413f650 2024-03-29T11:23:03.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.702+0000 7f2972ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:03.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.702+0000 7f2972ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:03.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.702+0000 7f2972ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:03.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.702+0000 7f2972ffd640 10 monclient(hunting): my global_id is 4314 2024-03-29T11:23:03.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.702+0000 7f2972ffd640 1 -- 172.21.15.86:0/3190058790 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2964004eb0 con 0x7f298413f650 2024-03-29T11:23:03.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.702+0000 7f2972ffd640 1 -- 172.21.15.86:0/3190058790 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1300339151 0 0) 0x7f29841dbee0 con 0x7f29841457f0 2024-03-29T11:23:03.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.702+0000 7f2972ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:03.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.702+0000 7f2972ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:03.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.702+0000 7f2972ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:03.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.702+0000 7f2972ffd640 10 monclient(hunting): my global_id is 4331 2024-03-29T11:23:03.682 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.702+0000 7f2972ffd640 1 -- 172.21.15.86:0/3190058790 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29841dfae0 con 0x7f29841457f0 2024-03-29T11:23:03.682 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.702+0000 7f2972ffd640 1 -- 172.21.15.86:0/3190058790 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1506934745 0 0) 0x7f2964004eb0 con 0x7f298413f650 2024-03-29T11:23:03.682 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.702+0000 7f2972ffd640 1 -- 172.21.15.86:0/3190058790 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29841dbee0 con 0x7f298413f650 2024-03-29T11:23:03.682 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.702+0000 7f2972ffd640 1 -- 172.21.15.86:0/3190058790 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f2978004080 con 0x7f298413f650 2024-03-29T11:23:03.682 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.702+0000 7f2972ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:03.682 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.702+0000 7f2972ffd640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:03.682 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.702+0000 7f2972ffd640 10 monclient(hunting): dump: 2024-03-29T11:23:03.682 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:03.682 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:03.682 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:03.682 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:03.682 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:03.682 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:03.682 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:03.683 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:03.683 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:03.683 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:03.683 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.702+0000 7f2972ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:03.683 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.702+0000 7f2972ffd640 1 -- 172.21.15.86:0/3190058790 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2938769457 0 0) 0x7f2984071110 con 0x7f29840a9d70 2024-03-29T11:23:03.683 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.702+0000 7f2972ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:03.683 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.702+0000 7f2972ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:03.683 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.702+0000 7f2972ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:03.683 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.702+0000 7f2972ffd640 10 monclient(hunting): my global_id is 4303 2024-03-29T11:23:03.683 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.702+0000 7f2972ffd640 1 -- 172.21.15.86:0/3190058790 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2964004eb0 con 0x7f29840a9d70 2024-03-29T11:23:03.683 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.702+0000 7f2972ffd640 1 -- 172.21.15.86:0/3190058790 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1629862410 0 0) 0x7f29841dfae0 con 0x7f29841457f0 2024-03-29T11:23:03.683 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.703+0000 7f2972ffd640 1 -- 172.21.15.86:0/3190058790 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2984071110 con 0x7f29841457f0 2024-03-29T11:23:03.683 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.703+0000 7f2972ffd640 1 -- 172.21.15.86:0/3190058790 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f297c001c30 con 0x7f29841457f0 2024-03-29T11:23:03.683 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.703+0000 7f2972ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:03.683 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.703+0000 7f2972ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:03.683 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.703+0000 7f2972ffd640 10 monclient(hunting): dump: 2024-03-29T11:23:03.683 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:03.684 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:03.684 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:03.684 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:03.684 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:03.684 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:03.684 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:03.684 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:03.684 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:03.684 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:03.687 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.703+0000 7f2972ffd640 1 -- 172.21.15.86:0/3190058790 <== mon.0 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3392115835 0 0) 0x7f29841dbee0 con 0x7f298413f650 2024-03-29T11:23:03.687 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.703+0000 7f2972ffd640 1 -- 172.21.15.86:0/3190058790 >> v1:172.21.15.86:6790/0 conn(0x7f29841457f0 legacy=0x7f29841df3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:03.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.703+0000 7f2972ffd640 1 -- 172.21.15.86:0/3190058790 >> v1:172.21.15.196:6789/0 conn(0x7f29840a9d70 legacy=0x7f29841decc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:03.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.703+0000 7f2972ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:23:03.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.703+0000 7f2972ffd640 1 monclient: found mon.a 2024-03-29T11:23:03.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.703+0000 7f2972ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:03.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.703+0000 7f2972ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:23:03.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.703+0000 7f2972ffd640 1 -- 172.21.15.86:0/3190058790 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29840b1770 con 0x7f298413f650 2024-03-29T11:23:03.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.703+0000 7f2972ffd640 10 monclient: _finish_auth 0 2024-03-29T11:23:03.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.703+0000 7f2972ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:23:03.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.703+0000 7f2972ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:03.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.704+0000 7f2972ffd640 1 -- 172.21.15.86:0/3190058790 <== mon.0 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2978000ed0 con 0x7f298413f650 2024-03-29T11:23:03.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.704+0000 7f2972ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:03.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.704+0000 7f2972ffd640 1 -- 172.21.15.86:0/3190058790 <== mon.0 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f29780012a0 con 0x7f298413f650 2024-03-29T11:23:03.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.704+0000 7f2972ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:03.689 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.704+0000 7f2972ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:03.689 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.704+0000 7f2972ffd640 10 monclient: dump: 2024-03-29T11:23:03.689 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:03.689 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:03.689 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:03.689 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:03.689 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:03.689 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:03.689 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:03.689 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:03.689 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:03.689 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:03.689 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.704+0000 7f2980d99640 4 set_mon_vals no callback set 2024-03-29T11:23:03.689 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.704+0000 7f2988d15640 5 monclient: authenticate success, global_id 4314 2024-03-29T11:23:03.689 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.704+0000 7f2988d15640 10 monclient: _renew_subs 2024-03-29T11:23:03.689 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.704+0000 7f2988d15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:23:03.690 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.704+0000 7f2988d15640 1 -- 172.21.15.86:0/3190058790 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29841dc400 con 0x7f298413f650 2024-03-29T11:23:03.690 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.705+0000 7f2988d15640 10 client.4314.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:03.690 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.705+0000 7f2988d15640 10 monclient: _renew_subs 2024-03-29T11:23:03.690 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.705+0000 7f2988d15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:23:03.690 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.705+0000 7f2988d15640 1 -- 172.21.15.86:0/3190058790 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2984071d60 con 0x7f298413f650 2024-03-29T11:23:03.690 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.705+0000 7f2988d15640 1 librados: init done 2024-03-29T11:23:03.690 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.706+0000 7f2988d15640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:03.690 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.706+0000 7f2988d15640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:03.690 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.706+0000 7f2988d15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:23:03.690 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.706+0000 7f2988d15640 1 -- 172.21.15.86:0/3190058790 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29841dc700 con 0x7f298413f650 2024-03-29T11:23:03.690 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.707+0000 7f2972ffd640 1 -- 172.21.15.86:0/3190058790 <== mon.0 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f2978001400 con 0x7f298413f650 2024-03-29T11:23:03.690 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.708+0000 7f2972ffd640 1 -- 172.21.15.86:0/3190058790 <== mon.0 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f29780849f0 con 0x7f298413f650 2024-03-29T11:23:03.690 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.708+0000 7f2972ffd640 10 client.4314.objecter ms_dispatch 0x7f2984000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:03.690 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.708+0000 7f2972ffd640 3 client.4314.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:03.691 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.708+0000 7f2972ffd640 3 client.4314.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:03.691 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.708+0000 7f2972ffd640 20 client.4314.objecter dump_active .. 0 homeless 2024-03-29T11:23:03.691 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.710+0000 7f298259c640 10 monclient: get_auth_request con 0x7f296406df70 auth_method 0 2024-03-29T11:23:03.691 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.711+0000 7f2972ffd640 10 client.4314.objecter ms_handle_connect 0x7f296406df70 2024-03-29T11:23:03.697 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.718+0000 7f2972ffd640 1 -- 172.21.15.86:0/3190058790 <== mon.0 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2978050330 con 0x7f298413f650 2024-03-29T11:23:03.697 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.718+0000 7f2972ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:03.697 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:03.718+0000 7f2972ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:04.019 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:04.038+0000 7f2988d15640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:04.019 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:04.038+0000 7f2988d15640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:04.019 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:04.038+0000 7f2988d15640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:23:04.019 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:04.038+0000 7f2988d15640 1 -- 172.21.15.86:0/3190058790 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29841dc070 con 0x7f298413f650 2024-03-29T11:23:04.020 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:04.042+0000 7f2972ffd640 1 -- 172.21.15.86:0/3190058790 <== mon.0 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f29780840a0 con 0x7f298413f650 2024-03-29T11:23:04.020 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:04.042+0000 7f2972ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:04.020 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:04.042+0000 7f2972ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:04.022 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:23:04.023 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:04.025 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:04.047+0000 7f2988d15640 10 librados: watch_flush enter 2024-03-29T11:23:04.028 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:04.050+0000 7f2988d15640 10 librados: watch_flush exit 2024-03-29T11:23:04.028 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:04.050+0000 7f2988d15640 20 client.4314.objecter shutdown clearing up homeless session... 2024-03-29T11:23:04.028 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:04.050+0000 7f2988d15640 10 client.4314.objecter successfully canceled tick 2024-03-29T11:23:04.028 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:04.050+0000 7f2988d15640 1 -- 172.21.15.86:0/3190058790 >> v1:172.21.15.196:6816/588110318 conn(0x7f296406df70 legacy=0x7f2964070430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:04.028 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:04.050+0000 7f2988d15640 10 monclient: shutdown 2024-03-29T11:23:04.028 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:04.050+0000 7f2988d15640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:04.028 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:04.050+0000 7f2988d15640 1 -- 172.21.15.86:0/3190058790 >> v1:172.21.15.86:6789/0 conn(0x7f298413f650 legacy=0x7f298406fda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:04.028 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:04.050+0000 7f2988d15640 1 -- 172.21.15.86:0/3190058790 shutdown_connections 2024-03-29T11:23:04.028 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:04.050+0000 7f2988d15640 1 -- 172.21.15.86:0/3190058790 >> 172.21.15.86:0/3190058790 conn(0x7f2984066a00 msgr2=0x7f298413ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:04.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:04.050+0000 7f2988d15640 1 -- 172.21.15.86:0/3190058790 shutdown_connections 2024-03-29T11:23:04.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:04.050+0000 7f2988d15640 1 -- 172.21.15.86:0/3190058790 wait complete. 2024-03-29T11:23:04.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:04.050+0000 7f2988d15640 1 librados: shutdown 2024-03-29T11:23:04.045 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:10.851 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:11.000 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.022+0000 7fec99f39640 10 monclient: get_monmap_and_config 2024-03-29T11:23:11.001 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.022+0000 7fec99f39640 10 monclient: build_initial_monmap 2024-03-29T11:23:11.001 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.022+0000 7fec99f39640 10 monclient: monmap: 2024-03-29T11:23:11.001 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:23:11.001 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:11.001 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:23:11.023727+0000 2024-03-29T11:23:11.001 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:23:11.023727+0000 2024-03-29T11:23:11.001 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:11.001 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:11.001 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:23:11.001 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:23:11.001 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:23:11.001 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:11.005 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.026+0000 7fec99f39640 1 Processor -- start 2024-03-29T11:23:11.005 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.027+0000 7fec99f39640 1 -- start start 2024-03-29T11:23:11.005 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.027+0000 7fec99f39640 10 monclient: init 2024-03-29T11:23:11.007 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.027+0000 7fec99f39640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:11.007 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.027+0000 7fec99f39640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:23:11.007 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.027+0000 7fec99f39640 10 monclient(hunting): picked mon.noname-b con 0x7fec941422b0 addr v1:172.21.15.86:6790/0 2024-03-29T11:23:11.008 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.027+0000 7fec99f39640 10 monclient(hunting): picked mon.noname-a con 0x7fec94142690 addr v1:172.21.15.86:6789/0 2024-03-29T11:23:11.008 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.027+0000 7fec99f39640 10 monclient(hunting): picked mon.noname-c con 0x7fec940b17d0 addr v1:172.21.15.196:6789/0 2024-03-29T11:23:11.008 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.027+0000 7fec99f39640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec94142c80 con 0x7fec94142690 2024-03-29T11:23:11.008 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.027+0000 7fec99f39640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec940aae40 con 0x7fec940b17d0 2024-03-29T11:23:11.008 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.027+0000 7fec99f39640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec9413fbd0 con 0x7fec941422b0 2024-03-29T11:23:11.008 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.027+0000 7fec99f39640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:11.008 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.027+0000 7fec99f39640 10 monclient(hunting): authenticate will time out at 1015.345946s 2024-03-29T11:23:11.008 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.030+0000 7fec92ffd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fec94142690 0x7fec940b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:60685/0 (socket says 172.21.15.86:60685) 2024-03-29T11:23:11.009 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.030+0000 7fec92ffd640 1 -- 172.21.15.86:0/365835146 learned_addr learned my addr 172.21.15.86:0/365835146 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:23:11.014 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.034+0000 7fec927fc640 1 -- 172.21.15.86:0/365835146 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1682843485 0 0) 0x7fec940aae40 con 0x7fec940b17d0 2024-03-29T11:23:11.015 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.034+0000 7fec927fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:11.015 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.034+0000 7fec927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:11.015 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.034+0000 7fec927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:11.015 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.034+0000 7fec927fc640 10 monclient(hunting): my global_id is 4306 2024-03-29T11:23:11.015 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.034+0000 7fec927fc640 1 -- 172.21.15.86:0/365835146 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec7c003520 con 0x7fec940b17d0 2024-03-29T11:23:11.015 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.034+0000 7fec927fc640 1 -- 172.21.15.86:0/365835146 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2602929237 0 0) 0x7fec94142c80 con 0x7fec94142690 2024-03-29T11:23:11.015 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.034+0000 7fec927fc640 10 monclient(hunting): _negotiate 2024-03-29T11:23:11.015 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.034+0000 7fec927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:11.015 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.034+0000 7fec927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:11.015 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.034+0000 7fec927fc640 10 monclient(hunting): my global_id is 4317 2024-03-29T11:23:11.015 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.034+0000 7fec927fc640 1 -- 172.21.15.86:0/365835146 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec940aae40 con 0x7fec94142690 2024-03-29T11:23:11.015 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.034+0000 7fec927fc640 1 -- 172.21.15.86:0/365835146 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3761808136 0 0) 0x7fec940aae40 con 0x7fec94142690 2024-03-29T11:23:11.015 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.034+0000 7fec927fc640 1 -- 172.21.15.86:0/365835146 >> v1:172.21.15.86:6790/0 conn(0x7fec941422b0 legacy=0x7fec9413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:11.016 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.034+0000 7fec927fc640 1 -- 172.21.15.86:0/365835146 >> v1:172.21.15.196:6789/0 conn(0x7fec940b17d0 legacy=0x7fec940b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:11.016 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.034+0000 7fec927fc640 10 monclient: _finish_hunting 0 2024-03-29T11:23:11.016 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.034+0000 7fec927fc640 1 monclient: found mon.noname-a 2024-03-29T11:23:11.016 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.034+0000 7fec927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:11.016 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.034+0000 7fec927fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.86:6789/0 2024-03-29T11:23:11.016 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.034+0000 7fec927fc640 1 -- 172.21.15.86:0/365835146 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec94151940 con 0x7fec94142690 2024-03-29T11:23:11.016 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.034+0000 7fec927fc640 10 monclient: _finish_auth 0 2024-03-29T11:23:11.017 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.034+0000 7fec927fc640 10 monclient: _check_auth_tickets 2024-03-29T11:23:11.017 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.034+0000 7fec927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:11.017 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.034+0000 7fec927fc640 1 -- 172.21.15.86:0/365835146 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fec84001520 con 0x7fec94142690 2024-03-29T11:23:11.017 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.034+0000 7fec927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:11.017 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.034+0000 7fec927fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:11.017 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.034+0000 7fec927fc640 10 monclient: dump: 2024-03-29T11:23:11.017 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:11.017 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:11.017 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:11.017 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:11.017 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:11.017 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:11.017 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:11.017 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:11.017 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:11.018 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:11.018 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.034+0000 7fec99f39640 5 monclient: authenticate success, global_id 4317 2024-03-29T11:23:11.018 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.034+0000 7fec99f39640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:11.018 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.035+0000 7fec927fc640 1 -- 172.21.15.86:0/365835146 <== mon.0 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec84001680 con 0x7fec94142690 2024-03-29T11:23:11.018 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.035+0000 7fec927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:11.018 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.035+0000 7fec927fc640 1 -- 172.21.15.86:0/365835146 <== mon.0 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fec84001a10 con 0x7fec94142690 2024-03-29T11:23:11.018 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.035+0000 7fec927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:11.018 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.035+0000 7fec927fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:11.018 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.035+0000 7fec927fc640 10 monclient: dump: 2024-03-29T11:23:11.018 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:11.018 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:11.018 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:11.018 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:11.018 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:11.019 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:11.019 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:11.019 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:11.019 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:11.019 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:11.019 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.035+0000 7fec99f39640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:11.019 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.035+0000 7fec99f39640 4 set_mon_vals no callback set 2024-03-29T11:23:11.019 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.035+0000 7fec99f39640 10 monclient: shutdown 2024-03-29T11:23:11.019 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.035+0000 7fec99f39640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:11.019 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.035+0000 7fec99f39640 1 -- 172.21.15.86:0/365835146 >> v1:172.21.15.86:6789/0 conn(0x7fec94142690 legacy=0x7fec940b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:11.019 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.037+0000 7fec99f39640 1 -- 172.21.15.86:0/365835146 shutdown_connections 2024-03-29T11:23:11.019 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.037+0000 7fec99f39640 1 -- 172.21.15.86:0/365835146 >> 172.21.15.86:0/365835146 conn(0x7fec94066a00 msgr2=0x7fec94066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:11.019 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.038+0000 7fec99f39640 1 -- 172.21.15.86:0/365835146 shutdown_connections 2024-03-29T11:23:11.019 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.038+0000 7fec99f39640 1 -- 172.21.15.86:0/365835146 wait complete. 2024-03-29T11:23:11.020 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.039+0000 7fec99f39640 10 monclient: build_initial_monmap 2024-03-29T11:23:11.020 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.039+0000 7fec99f39640 10 monclient: monmap: 2024-03-29T11:23:11.021 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:23:11.021 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:11.021 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:23:11.021 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:23:11.021 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:11.021 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:11.021 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:23:11.021 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:23:11.021 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:23:11.021 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:11.021 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.039+0000 7fec99f39640 1 librados: starting msgr at 2024-03-29T11:23:11.021 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.039+0000 7fec99f39640 1 librados: starting objecter 2024-03-29T11:23:11.021 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.040+0000 7fec99f39640 1 Processor -- start 2024-03-29T11:23:11.021 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.040+0000 7fec99f39640 1 -- start start 2024-03-29T11:23:11.021 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.040+0000 7fec99f39640 1 librados: setting wanted keys 2024-03-29T11:23:11.022 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.040+0000 7fec99f39640 1 librados: calling monclient init 2024-03-29T11:23:11.022 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.040+0000 7fec99f39640 10 monclient: init 2024-03-29T11:23:11.022 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.041+0000 7fec99f39640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:11.022 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.041+0000 7fec99f39640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:23:11.022 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.041+0000 7fec99f39640 10 monclient(hunting): picked mon.noname-c con 0x7fec940b17d0 addr v1:172.21.15.86:6790/0 2024-03-29T11:23:11.022 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.041+0000 7fec99f39640 10 monclient(hunting): picked mon.noname-b con 0x7fec941422b0 addr v1:172.21.15.196:6789/0 2024-03-29T11:23:11.022 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.041+0000 7fec937fe640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7fec940b17d0 0x7fec9406f1a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:60838/0 (socket says 172.21.15.86:60838) 2024-03-29T11:23:11.026 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.043+0000 7fec99f39640 10 monclient(hunting): picked mon.noname-a con 0x7fec94142690 addr v1:172.21.15.86:6789/0 2024-03-29T11:23:11.026 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.043+0000 7fec99f39640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec940ac550 con 0x7fec94142690 2024-03-29T11:23:11.026 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.043+0000 7fec99f39640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec940ac7c0 con 0x7fec941422b0 2024-03-29T11:23:11.026 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.043+0000 7fec99f39640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec940acaa0 con 0x7fec940b17d0 2024-03-29T11:23:11.026 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.043+0000 7fec99f39640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:11.026 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.043+0000 7fec99f39640 10 monclient(hunting): authenticate will time out at 1015.361610s 2024-03-29T11:23:11.026 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.043+0000 7fec937fe640 1 -- 172.21.15.86:0/2210871337 learned_addr learned my addr 172.21.15.86:0/2210871337 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:23:11.026 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.044+0000 7fec7bfff640 10 client.?.objecter ms_handle_connect 0x7fec940b17d0 2024-03-29T11:23:11.026 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.044+0000 7fec7bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:11.026 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.044+0000 7fec7bfff640 10 client.?.objecter ms_handle_connect 0x7fec94142690 2024-03-29T11:23:11.026 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.044+0000 7fec7bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:11.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.044+0000 7fec7bfff640 1 -- 172.21.15.86:0/2210871337 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 111244887 0 0) 0x7fec940acaa0 con 0x7fec940b17d0 2024-03-29T11:23:11.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.044+0000 7fec7bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:11.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.044+0000 7fec7bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:11.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.044+0000 7fec7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:11.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.044+0000 7fec7bfff640 10 monclient(hunting): my global_id is 4337 2024-03-29T11:23:11.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.044+0000 7fec7bfff640 1 -- 172.21.15.86:0/2210871337 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec80004eb0 con 0x7fec940b17d0 2024-03-29T11:23:11.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.045+0000 7fec7bfff640 1 -- 172.21.15.86:0/2210871337 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2564638585 0 0) 0x7fec940ac550 con 0x7fec94142690 2024-03-29T11:23:11.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.045+0000 7fec7bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:23:11.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.045+0000 7fec7bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:11.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.045+0000 7fec7bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:11.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.045+0000 7fec7bfff640 10 monclient(hunting): my global_id is 4320 2024-03-29T11:23:11.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.045+0000 7fec7bfff640 1 -- 172.21.15.86:0/2210871337 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec940acaa0 con 0x7fec94142690 2024-03-29T11:23:11.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.045+0000 7fec7bfff640 1 -- 172.21.15.86:0/2210871337 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3751384424 0 0) 0x7fec940acaa0 con 0x7fec94142690 2024-03-29T11:23:11.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.045+0000 7fec7bfff640 1 -- 172.21.15.86:0/2210871337 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec940ac550 con 0x7fec94142690 2024-03-29T11:23:11.028 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.045+0000 7fec7bfff640 1 -- 172.21.15.86:0/2210871337 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fec8c001be0 con 0x7fec94142690 2024-03-29T11:23:11.028 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.046+0000 7fec7bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:11.028 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.046+0000 7fec7bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:11.028 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.046+0000 7fec7bfff640 10 monclient(hunting): dump: 2024-03-29T11:23:11.028 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:11.028 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:11.028 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:11.028 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:11.028 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:11.028 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:11.028 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:11.028 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:11.028 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:11.028 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:11.028 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.046+0000 7fec7bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:11.028 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.046+0000 7fec7bfff640 1 -- 172.21.15.86:0/2210871337 <== mon.0 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1128434120 0 0) 0x7fec940ac550 con 0x7fec94142690 2024-03-29T11:23:11.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.046+0000 7fec7bfff640 1 -- 172.21.15.86:0/2210871337 >> v1:172.21.15.86:6790/0 conn(0x7fec940b17d0 legacy=0x7fec9406f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:11.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.046+0000 7fec7bfff640 1 -- 172.21.15.86:0/2210871337 >> v1:172.21.15.196:6789/0 conn(0x7fec941422b0 legacy=0x7fec9406f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:11.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.046+0000 7fec7bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:23:11.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.046+0000 7fec7bfff640 1 monclient: found mon.a 2024-03-29T11:23:11.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.046+0000 7fec7bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:11.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.046+0000 7fec7bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:23:11.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.046+0000 7fec7bfff640 1 -- 172.21.15.86:0/2210871337 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec940acd80 con 0x7fec94142690 2024-03-29T11:23:11.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.047+0000 7fec7bfff640 10 monclient: _finish_auth 0 2024-03-29T11:23:11.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.047+0000 7fec7bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:23:11.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.047+0000 7fec7bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:11.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.047+0000 7fec7bfff640 1 -- 172.21.15.86:0/2210871337 <== mon.0 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec8c003990 con 0x7fec94142690 2024-03-29T11:23:11.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.047+0000 7fec99f39640 5 monclient: authenticate success, global_id 4320 2024-03-29T11:23:11.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.047+0000 7fec99f39640 10 monclient: _renew_subs 2024-03-29T11:23:11.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.047+0000 7fec99f39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:23:11.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.047+0000 7fec99f39640 1 -- 172.21.15.86:0/2210871337 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec940713f0 con 0x7fec94142690 2024-03-29T11:23:11.030 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.047+0000 7fec99f39640 10 client.4320.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:11.030 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.047+0000 7fec7bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:11.030 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.047+0000 7fec99f39640 10 monclient: _renew_subs 2024-03-29T11:23:11.030 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.047+0000 7fec99f39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:23:11.030 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.047+0000 7fec99f39640 1 -- 172.21.15.86:0/2210871337 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fec940ab7b0 con 0x7fec94142690 2024-03-29T11:23:11.030 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.047+0000 7fec91ffb640 4 set_mon_vals no callback set 2024-03-29T11:23:11.030 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.047+0000 7fec99f39640 1 librados: init done 2024-03-29T11:23:11.030 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.047+0000 7fec7bfff640 1 -- 172.21.15.86:0/2210871337 <== mon.0 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fec8c003ca0 con 0x7fec94142690 2024-03-29T11:23:11.030 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.047+0000 7fec7bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:11.030 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.047+0000 7fec7bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:11.030 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.047+0000 7fec7bfff640 10 monclient: dump: 2024-03-29T11:23:11.030 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:11.030 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:11.030 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:11.030 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:11.031 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:11.031 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:11.031 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:11.031 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:11.031 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:11.031 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:11.032 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.049+0000 7fec7bfff640 1 -- 172.21.15.86:0/2210871337 <== mon.0 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7fec8c019c70 con 0x7fec94142690 2024-03-29T11:23:11.032 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.049+0000 7fec7bfff640 1 -- 172.21.15.86:0/2210871337 <== mon.0 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7fec8c083af0 con 0x7fec94142690 2024-03-29T11:23:11.032 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.049+0000 7fec7bfff640 10 client.4320.objecter ms_dispatch 0x7fec94000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:11.032 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.049+0000 7fec7bfff640 3 client.4320.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:11.033 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.049+0000 7fec7bfff640 3 client.4320.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:11.033 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.050+0000 7fec937fe640 10 monclient: get_auth_request con 0x7fec8006daf0 auth_method 0 2024-03-29T11:23:11.033 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.050+0000 7fec7bfff640 20 client.4320.objecter dump_active .. 0 homeless 2024-03-29T11:23:11.033 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.050+0000 7fec99f39640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:11.033 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.051+0000 7fec99f39640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:11.033 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.051+0000 7fec99f39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:23:11.033 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.051+0000 7fec99f39640 1 -- 172.21.15.86:0/2210871337 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec94071250 con 0x7fec94142690 2024-03-29T11:23:11.033 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.054+0000 7fec7bfff640 10 client.4320.objecter ms_handle_connect 0x7fec8006daf0 2024-03-29T11:23:11.038 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.059+0000 7fec7bfff640 1 -- 172.21.15.86:0/2210871337 <== mon.0 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fec94071250 con 0x7fec94142690 2024-03-29T11:23:11.038 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.059+0000 7fec7bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:11.038 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.059+0000 7fec7bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:11.369 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.389+0000 7fec99f39640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:11.369 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.389+0000 7fec99f39640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:11.369 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.389+0000 7fec99f39640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:23:11.369 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.389+0000 7fec99f39640 1 -- 172.21.15.86:0/2210871337 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec94071250 con 0x7fec94142690 2024-03-29T11:23:11.369 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.390+0000 7fec7bfff640 1 -- 172.21.15.86:0/2210871337 <== mon.0 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7fec94071250 con 0x7fec94142690 2024-03-29T11:23:11.369 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.390+0000 7fec7bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:11.370 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.390+0000 7fec7bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:11.371 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:23:11.371 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:11.374 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.394+0000 7fec99f39640 10 librados: watch_flush enter 2024-03-29T11:23:11.374 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.394+0000 7fec99f39640 10 librados: watch_flush exit 2024-03-29T11:23:11.374 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.394+0000 7fec99f39640 20 client.4320.objecter shutdown clearing up homeless session... 2024-03-29T11:23:11.374 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.394+0000 7fec99f39640 10 client.4320.objecter successfully canceled tick 2024-03-29T11:23:11.374 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.394+0000 7fec99f39640 1 -- 172.21.15.86:0/2210871337 >> v1:172.21.15.196:6816/588110318 conn(0x7fec8006daf0 legacy=0x7fec8006ffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:11.374 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.394+0000 7fec99f39640 10 monclient: shutdown 2024-03-29T11:23:11.374 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.394+0000 7fec99f39640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:11.374 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.394+0000 7fec99f39640 1 -- 172.21.15.86:0/2210871337 >> v1:172.21.15.86:6789/0 conn(0x7fec94142690 legacy=0x7fec941dda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:11.374 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.394+0000 7fec99f39640 1 -- 172.21.15.86:0/2210871337 shutdown_connections 2024-03-29T11:23:11.374 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.394+0000 7fec99f39640 1 -- 172.21.15.86:0/2210871337 >> 172.21.15.86:0/2210871337 conn(0x7fec94066a00 msgr2=0x7fec9413db00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:11.375 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.395+0000 7fec99f39640 1 -- 172.21.15.86:0/2210871337 shutdown_connections 2024-03-29T11:23:11.375 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.395+0000 7fec99f39640 1 -- 172.21.15.86:0/2210871337 wait complete. 2024-03-29T11:23:11.375 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:11.395+0000 7fec99f39640 1 librados: shutdown 2024-03-29T11:23:11.393 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:18.200 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:18.345 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.366+0000 7fea710d9640 10 monclient: get_monmap_and_config 2024-03-29T11:23:18.345 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.366+0000 7fea710d9640 10 monclient: build_initial_monmap 2024-03-29T11:23:18.345 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.366+0000 7fea710d9640 10 monclient: monmap: 2024-03-29T11:23:18.345 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:23:18.346 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:18.346 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:23:18.367871+0000 2024-03-29T11:23:18.346 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:23:18.367871+0000 2024-03-29T11:23:18.346 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:18.346 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:18.346 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:23:18.346 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:23:18.346 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:23:18.346 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:18.347 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.367+0000 7fea710d9640 1 Processor -- start 2024-03-29T11:23:18.347 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.368+0000 7fea710d9640 1 -- start start 2024-03-29T11:23:18.347 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.368+0000 7fea710d9640 10 monclient: init 2024-03-29T11:23:18.347 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.368+0000 7fea710d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:18.347 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.368+0000 7fea710d9640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:23:18.347 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.369+0000 7fea710d9640 10 monclient(hunting): picked mon.noname-c con 0x7fea64038c60 addr v1:172.21.15.196:6789/0 2024-03-29T11:23:18.347 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.369+0000 7fea710d9640 10 monclient(hunting): picked mon.noname-b con 0x7fea640324b0 addr v1:172.21.15.86:6790/0 2024-03-29T11:23:18.347 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.369+0000 7fea710d9640 10 monclient(hunting): picked mon.noname-a con 0x7fea640d11e0 addr v1:172.21.15.86:6789/0 2024-03-29T11:23:18.347 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.369+0000 7fea710d9640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea640caf80 con 0x7fea640d11e0 2024-03-29T11:23:18.348 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.369+0000 7fea710d9640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea64039850 con 0x7fea64038c60 2024-03-29T11:23:18.348 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.369+0000 7fea710d9640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea640c78d0 con 0x7fea640324b0 2024-03-29T11:23:18.348 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.369+0000 7fea710d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:18.348 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.369+0000 7fea710d9640 10 monclient(hunting): authenticate will time out at 1022.687062s 2024-03-29T11:23:18.348 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.369+0000 7fea6b7fe640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7fea640324b0 0x7fea640cb4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:34920/0 (socket says 172.21.15.86:34920) 2024-03-29T11:23:18.348 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.369+0000 7fea6b7fe640 1 -- 172.21.15.86:0/125918059 learned_addr learned my addr 172.21.15.86:0/125918059 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:23:18.354 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.370+0000 7fea6affd640 1 -- 172.21.15.86:0/125918059 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2658885969 0 0) 0x7fea640c78d0 con 0x7fea640324b0 2024-03-29T11:23:18.355 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.370+0000 7fea6affd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:18.355 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.370+0000 7fea6affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:18.355 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.370+0000 7fea6affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:18.355 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.370+0000 7fea6affd640 10 monclient(hunting): my global_id is 4340 2024-03-29T11:23:18.355 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.370+0000 7fea6affd640 1 -- 172.21.15.86:0/125918059 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea54003520 con 0x7fea640324b0 2024-03-29T11:23:18.355 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.371+0000 7fea6affd640 1 -- 172.21.15.86:0/125918059 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 5660297 0 0) 0x7fea54003520 con 0x7fea640324b0 2024-03-29T11:23:18.355 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.371+0000 7fea6affd640 1 -- 172.21.15.86:0/125918059 >> v1:172.21.15.196:6789/0 conn(0x7fea64038c60 legacy=0x7fea640c4ea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:18.355 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.372+0000 7fea6affd640 1 -- 172.21.15.86:0/125918059 >> v1:172.21.15.86:6789/0 conn(0x7fea640d11e0 legacy=0x7fea640d35d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:18.355 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.372+0000 7fea6affd640 10 monclient: _finish_hunting 0 2024-03-29T11:23:18.355 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.372+0000 7fea6affd640 1 monclient: found mon.noname-b 2024-03-29T11:23:18.355 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.372+0000 7fea6affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:18.355 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.372+0000 7fea6affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:23:18.355 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.372+0000 7fea6affd640 1 -- 172.21.15.86:0/125918059 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea640c7a60 con 0x7fea640324b0 2024-03-29T11:23:18.355 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.372+0000 7fea6affd640 10 monclient: _finish_auth 0 2024-03-29T11:23:18.356 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.372+0000 7fea6affd640 10 monclient: _check_auth_tickets 2024-03-29T11:23:18.356 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.372+0000 7fea6affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:18.356 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.372+0000 7fea710d9640 5 monclient: authenticate success, global_id 4340 2024-03-29T11:23:18.356 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.372+0000 7fea710d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:18.356 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.373+0000 7fea6affd640 1 -- 172.21.15.86:0/125918059 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fea60001690 con 0x7fea640324b0 2024-03-29T11:23:18.356 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.373+0000 7fea6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:18.356 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.373+0000 7fea6affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:18.356 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.373+0000 7fea6affd640 10 monclient: dump: 2024-03-29T11:23:18.356 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:18.356 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:18.356 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:18.356 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:18.356 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:18.356 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:18.356 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:18.357 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:18.357 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:18.357 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:18.357 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.373+0000 7fea710d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:18.357 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.373+0000 7fea6affd640 1 -- 172.21.15.86:0/125918059 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea60001920 con 0x7fea640324b0 2024-03-29T11:23:18.357 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.373+0000 7fea6affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:18.357 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.373+0000 7fea710d9640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:18.357 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.373+0000 7fea710d9640 4 set_mon_vals no callback set 2024-03-29T11:23:18.357 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.374+0000 7fea6affd640 1 -- 172.21.15.86:0/125918059 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fea60001cb0 con 0x7fea640324b0 2024-03-29T11:23:18.357 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.374+0000 7fea6affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:18.357 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.374+0000 7fea6affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:18.357 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.374+0000 7fea6affd640 10 monclient: dump: 2024-03-29T11:23:18.357 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:18.357 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:18.357 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:18.358 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:18.358 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:18.358 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:18.358 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:18.358 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:18.358 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:18.358 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:18.358 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.375+0000 7fea710d9640 10 monclient: shutdown 2024-03-29T11:23:18.358 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.375+0000 7fea710d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:18.358 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.375+0000 7fea710d9640 1 -- 172.21.15.86:0/125918059 >> v1:172.21.15.86:6790/0 conn(0x7fea640324b0 legacy=0x7fea640cb4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:18.358 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.375+0000 7fea710d9640 1 -- 172.21.15.86:0/125918059 shutdown_connections 2024-03-29T11:23:18.358 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.375+0000 7fea710d9640 1 -- 172.21.15.86:0/125918059 >> 172.21.15.86:0/125918059 conn(0x7fea64005c20 msgr2=0x7fea64006030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:18.358 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.376+0000 7fea710d9640 1 -- 172.21.15.86:0/125918059 shutdown_connections 2024-03-29T11:23:18.358 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.376+0000 7fea710d9640 1 -- 172.21.15.86:0/125918059 wait complete. 2024-03-29T11:23:18.360 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.377+0000 7fea710d9640 10 monclient: build_initial_monmap 2024-03-29T11:23:18.360 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.377+0000 7fea710d9640 10 monclient: monmap: 2024-03-29T11:23:18.360 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:23:18.360 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:18.360 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:23:18.360 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:23:18.360 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:18.360 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:18.360 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:23:18.360 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:23:18.361 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:23:18.361 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:18.361 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.377+0000 7fea710d9640 1 librados: starting msgr at 2024-03-29T11:23:18.361 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.377+0000 7fea710d9640 1 librados: starting objecter 2024-03-29T11:23:18.361 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.377+0000 7fea710d9640 1 Processor -- start 2024-03-29T11:23:18.361 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.378+0000 7fea710d9640 1 -- start start 2024-03-29T11:23:18.361 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.378+0000 7fea710d9640 1 librados: setting wanted keys 2024-03-29T11:23:18.361 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.378+0000 7fea710d9640 1 librados: calling monclient init 2024-03-29T11:23:18.361 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.378+0000 7fea710d9640 10 monclient: init 2024-03-29T11:23:18.361 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.380+0000 7fea710d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:18.361 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.380+0000 7fea710d9640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:18.361 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.380+0000 7fea710d9640 10 monclient(hunting): picked mon.noname-a con 0x7fea640324b0 addr v1:172.21.15.86:6789/0 2024-03-29T11:23:18.361 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.380+0000 7fea710d9640 10 monclient(hunting): picked mon.noname-c con 0x7fea64038c60 addr v1:172.21.15.86:6790/0 2024-03-29T11:23:18.361 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.380+0000 7fea710d9640 10 monclient(hunting): picked mon.noname-b con 0x7fea640d11e0 addr v1:172.21.15.196:6789/0 2024-03-29T11:23:18.361 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.380+0000 7fea710d9640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea640bc950 con 0x7fea640324b0 2024-03-29T11:23:18.362 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.380+0000 7fea710d9640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea6400a230 con 0x7fea640d11e0 2024-03-29T11:23:18.362 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.380+0000 7fea710d9640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea64030030 con 0x7fea64038c60 2024-03-29T11:23:18.362 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.380+0000 7fea710d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:18.362 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.380+0000 7fea710d9640 10 monclient(hunting): authenticate will time out at 1022.698299s 2024-03-29T11:23:18.362 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.380+0000 7fea6b7fe640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7fea64038c60 0x7fea6402d7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:37432/0 (socket says 172.21.15.86:37432) 2024-03-29T11:23:18.362 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.380+0000 7fea6b7fe640 1 -- 172.21.15.86:0/15539786 learned_addr learned my addr 172.21.15.86:0/15539786 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:23:18.366 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.380+0000 7fea68ff9640 10 client.?.objecter ms_handle_connect 0x7fea64038c60 2024-03-29T11:23:18.366 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.380+0000 7fea68ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:18.366 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.383+0000 7fea68ff9640 1 -- 172.21.15.86:0/15539786 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1407548262 0 0) 0x7fea64030030 con 0x7fea64038c60 2024-03-29T11:23:18.367 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.384+0000 7fea68ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:23:18.367 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.384+0000 7fea68ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:18.367 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.384+0000 7fea68ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:18.367 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.384+0000 7fea68ff9640 10 monclient(hunting): my global_id is 4343 2024-03-29T11:23:18.367 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.384+0000 7fea68ff9640 1 -- 172.21.15.86:0/15539786 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea58004eb0 con 0x7fea64038c60 2024-03-29T11:23:18.367 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.384+0000 7fea68ff9640 10 client.?.objecter ms_handle_connect 0x7fea640324b0 2024-03-29T11:23:18.367 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.384+0000 7fea68ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:18.367 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.384+0000 7fea68ff9640 10 client.?.objecter ms_handle_connect 0x7fea640d11e0 2024-03-29T11:23:18.367 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.384+0000 7fea68ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:18.367 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.384+0000 7fea68ff9640 1 -- 172.21.15.86:0/15539786 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3157830248 0 0) 0x7fea640bc950 con 0x7fea640324b0 2024-03-29T11:23:18.367 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.384+0000 7fea68ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:23:18.367 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.384+0000 7fea68ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:18.367 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.384+0000 7fea68ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:18.367 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.384+0000 7fea68ff9640 10 monclient(hunting): my global_id is 4323 2024-03-29T11:23:18.368 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.384+0000 7fea68ff9640 1 -- 172.21.15.86:0/15539786 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fea64030030 con 0x7fea640324b0 2024-03-29T11:23:18.368 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.384+0000 7fea68ff9640 1 -- 172.21.15.86:0/15539786 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1352204292 0 0) 0x7fea58004eb0 con 0x7fea64038c60 2024-03-29T11:23:18.368 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.384+0000 7fea68ff9640 1 -- 172.21.15.86:0/15539786 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fea640bc950 con 0x7fea64038c60 2024-03-29T11:23:18.368 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.384+0000 7fea68ff9640 1 -- 172.21.15.86:0/15539786 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fea60001610 con 0x7fea64038c60 2024-03-29T11:23:18.368 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.384+0000 7fea68ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:18.368 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.384+0000 7fea68ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:18.368 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.384+0000 7fea68ff9640 10 monclient(hunting): dump: 2024-03-29T11:23:18.368 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:18.368 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:18.368 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:18.368 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:18.368 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:18.368 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:18.368 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:18.369 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:18.369 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:18.369 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:18.369 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.384+0000 7fea68ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:18.369 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.384+0000 7fea68ff9640 1 -- 172.21.15.86:0/15539786 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 967307423 0 0) 0x7fea640bc950 con 0x7fea64038c60 2024-03-29T11:23:18.369 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.385+0000 7fea68ff9640 1 -- 172.21.15.86:0/15539786 >> v1:172.21.15.196:6789/0 conn(0x7fea640d11e0 legacy=0x7fea64163a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:18.369 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.385+0000 7fea68ff9640 1 -- 172.21.15.86:0/15539786 >> v1:172.21.15.86:6789/0 conn(0x7fea640324b0 legacy=0x7fea6402f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:18.369 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.385+0000 7fea68ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:23:18.369 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.385+0000 7fea68ff9640 1 monclient: found mon.c 2024-03-29T11:23:18.369 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.385+0000 7fea68ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:18.369 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.385+0000 7fea68ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:23:18.369 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.385+0000 7fea68ff9640 1 -- 172.21.15.86:0/15539786 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea640301c0 con 0x7fea64038c60 2024-03-29T11:23:18.369 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.385+0000 7fea68ff9640 10 monclient: _finish_auth 0 2024-03-29T11:23:18.369 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.385+0000 7fea68ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:23:18.369 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.385+0000 7fea68ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:18.370 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.385+0000 7fea710d9640 5 monclient: authenticate success, global_id 4343 2024-03-29T11:23:18.370 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.385+0000 7fea710d9640 10 monclient: _renew_subs 2024-03-29T11:23:18.370 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.385+0000 7fea710d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:23:18.370 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.385+0000 7fea710d9640 1 -- 172.21.15.86:0/15539786 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea6402ec10 con 0x7fea64038c60 2024-03-29T11:23:18.370 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.385+0000 7fea710d9640 10 client.4343.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:18.370 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.385+0000 7fea710d9640 10 monclient: _renew_subs 2024-03-29T11:23:18.370 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.385+0000 7fea710d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:23:18.370 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.385+0000 7fea710d9640 1 -- 172.21.15.86:0/15539786 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fea640bc650 con 0x7fea64038c60 2024-03-29T11:23:18.370 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.385+0000 7fea710d9640 1 librados: init done 2024-03-29T11:23:18.370 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.385+0000 7fea68ff9640 1 -- 172.21.15.86:0/15539786 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fea60001770 con 0x7fea64038c60 2024-03-29T11:23:18.370 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.385+0000 7fea68ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:18.370 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.385+0000 7fea68ff9640 1 -- 172.21.15.86:0/15539786 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fea60004d90 con 0x7fea64038c60 2024-03-29T11:23:18.370 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.385+0000 7fea68ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:18.370 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.385+0000 7fea68ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:18.371 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.385+0000 7fea68ff9640 10 monclient: dump: 2024-03-29T11:23:18.371 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:18.371 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:18.371 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:18.371 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:18.371 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:18.371 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:18.371 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:18.371 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:18.371 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:18.371 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:18.371 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.387+0000 7fea6a7fc640 4 set_mon_vals no callback set 2024-03-29T11:23:18.371 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.388+0000 7fea68ff9640 1 -- 172.21.15.86:0/15539786 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7fea60004ef0 con 0x7fea64038c60 2024-03-29T11:23:18.373 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.388+0000 7fea68ff9640 1 -- 172.21.15.86:0/15539786 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7fea60084c80 con 0x7fea64038c60 2024-03-29T11:23:18.373 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.388+0000 7fea68ff9640 10 client.4343.objecter ms_dispatch 0x7fea6c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:18.373 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.388+0000 7fea68ff9640 3 client.4343.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:18.373 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.388+0000 7fea710d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:18.373 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.388+0000 7fea710d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:18.373 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.388+0000 7fea710d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:23:18.373 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.388+0000 7fea710d9640 1 -- 172.21.15.86:0/15539786 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea4c005080 con 0x7fea64038c60 2024-03-29T11:23:18.373 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.389+0000 7fea68ff9640 3 client.4343.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:18.373 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.389+0000 7fea6bfff640 10 monclient: get_auth_request con 0x7fea5806dcc0 auth_method 0 2024-03-29T11:23:18.373 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.389+0000 7fea68ff9640 20 client.4343.objecter dump_active .. 0 homeless 2024-03-29T11:23:18.373 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.389+0000 7fea68ff9640 10 client.4343.objecter ms_handle_connect 0x7fea5806dcc0 2024-03-29T11:23:18.376 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.397+0000 7fea68ff9640 1 -- 172.21.15.86:0/15539786 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fea60084e60 con 0x7fea64038c60 2024-03-29T11:23:18.376 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.397+0000 7fea68ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:18.376 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.397+0000 7fea68ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:18.696 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.718+0000 7fea710d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:18.697 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.718+0000 7fea710d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:18.697 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.718+0000 7fea710d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:23:18.697 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.718+0000 7fea710d9640 1 -- 172.21.15.86:0/15539786 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea4c005330 con 0x7fea64038c60 2024-03-29T11:23:18.697 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.719+0000 7fea68ff9640 1 -- 172.21.15.86:0/15539786 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7fea60054120 con 0x7fea64038c60 2024-03-29T11:23:18.697 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.719+0000 7fea68ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:18.697 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.719+0000 7fea68ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:18.698 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:23:18.698 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:18.701 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.722+0000 7fea710d9640 10 librados: watch_flush enter 2024-03-29T11:23:18.701 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.722+0000 7fea710d9640 10 librados: watch_flush exit 2024-03-29T11:23:18.701 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.722+0000 7fea710d9640 20 client.4343.objecter shutdown clearing up homeless session... 2024-03-29T11:23:18.701 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.722+0000 7fea710d9640 10 client.4343.objecter successfully canceled tick 2024-03-29T11:23:18.702 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.723+0000 7fea710d9640 1 -- 172.21.15.86:0/15539786 >> v1:172.21.15.196:6816/588110318 conn(0x7fea5806dcc0 legacy=0x7fea58070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:18.702 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.723+0000 7fea710d9640 10 monclient: shutdown 2024-03-29T11:23:18.702 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.723+0000 7fea710d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:18.702 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.723+0000 7fea710d9640 1 -- 172.21.15.86:0/15539786 >> v1:172.21.15.86:6790/0 conn(0x7fea64038c60 legacy=0x7fea6402d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:18.702 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.723+0000 7fea710d9640 1 -- 172.21.15.86:0/15539786 shutdown_connections 2024-03-29T11:23:18.702 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.723+0000 7fea710d9640 1 -- 172.21.15.86:0/15539786 >> 172.21.15.86:0/15539786 conn(0x7fea64005c20 msgr2=0x7fea640043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:18.703 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.724+0000 7fea710d9640 1 -- 172.21.15.86:0/15539786 shutdown_connections 2024-03-29T11:23:18.703 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.724+0000 7fea710d9640 1 -- 172.21.15.86:0/15539786 wait complete. 2024-03-29T11:23:18.703 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:18.724+0000 7fea710d9640 1 librados: shutdown 2024-03-29T11:23:18.720 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:25.525 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:25.683 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.704+0000 7fae124d3640 10 monclient: get_monmap_and_config 2024-03-29T11:23:25.684 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.704+0000 7fae124d3640 10 monclient: build_initial_monmap 2024-03-29T11:23:25.684 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.704+0000 7fae124d3640 10 monclient: monmap: 2024-03-29T11:23:25.684 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:23:25.684 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:25.684 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:23:25.705773+0000 2024-03-29T11:23:25.684 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:23:25.705773+0000 2024-03-29T11:23:25.684 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:25.684 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:25.684 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:23:25.684 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:23:25.684 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:23:25.684 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:25.685 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.706+0000 7fae124d3640 1 Processor -- start 2024-03-29T11:23:25.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.709+0000 7fae124d3640 1 -- start start 2024-03-29T11:23:25.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.709+0000 7fae124d3640 10 monclient: init 2024-03-29T11:23:25.690 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.710+0000 7fae124d3640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:25.690 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.710+0000 7fae124d3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:25.690 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.711+0000 7fae124d3640 10 monclient(hunting): picked mon.noname-c con 0x7fae0c1457f0 addr v1:172.21.15.196:6789/0 2024-03-29T11:23:25.691 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.711+0000 7fae124d3640 10 monclient(hunting): picked mon.noname-a con 0x7fae0c13f650 addr v1:172.21.15.86:6789/0 2024-03-29T11:23:25.691 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.711+0000 7fae124d3640 10 monclient(hunting): picked mon.noname-b con 0x7fae0c0a9d70 addr v1:172.21.15.86:6790/0 2024-03-29T11:23:25.691 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.711+0000 7fae124d3640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae0c0aaef0 con 0x7fae0c13f650 2024-03-29T11:23:25.691 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.711+0000 7fae124d3640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae0c145bd0 con 0x7fae0c1457f0 2024-03-29T11:23:25.691 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.711+0000 7fae124d3640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae0c145eb0 con 0x7fae0c0a9d70 2024-03-29T11:23:25.691 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.711+0000 7fae124d3640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:25.691 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.711+0000 7fae124d3640 10 monclient(hunting): authenticate will time out at 1030.029092s 2024-03-29T11:23:25.694 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.714+0000 7fae10a70640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7fae0c0a9d70 0x7fae0c14d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:44863/0 (socket says 172.21.15.86:44863) 2024-03-29T11:23:25.694 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.714+0000 7fae10a70640 1 -- 172.21.15.86:0/2511663031 learned_addr learned my addr 172.21.15.86:0/2511663031 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:23:25.694 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.715+0000 7fae0affd640 1 -- 172.21.15.86:0/2511663031 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4062085989 0 0) 0x7fae0c145eb0 con 0x7fae0c0a9d70 2024-03-29T11:23:25.694 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.715+0000 7fae0affd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:25.695 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.715+0000 7fae0affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:25.695 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.715+0000 7fae0affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:25.695 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.715+0000 7fae0affd640 10 monclient(hunting): my global_id is 4346 2024-03-29T11:23:25.695 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.715+0000 7fae0affd640 1 -- 172.21.15.86:0/2511663031 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadf4003520 con 0x7fae0c0a9d70 2024-03-29T11:23:25.696 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.717+0000 7fae0affd640 1 -- 172.21.15.86:0/2511663031 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 418681655 0 0) 0x7fadf4003520 con 0x7fae0c0a9d70 2024-03-29T11:23:25.696 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.717+0000 7fae0affd640 1 -- 172.21.15.86:0/2511663031 >> v1:172.21.15.196:6789/0 conn(0x7fae0c1457f0 legacy=0x7fae0c13ef40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:23:25.696 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.717+0000 7fae0affd640 1 -- 172.21.15.86:0/2511663031 >> v1:172.21.15.86:6789/0 conn(0x7fae0c13f650 legacy=0x7fae0c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:25.696 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.717+0000 7fae0affd640 10 monclient: _finish_hunting 0 2024-03-29T11:23:25.696 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.717+0000 7fae0affd640 1 monclient: found mon.noname-b 2024-03-29T11:23:25.696 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.717+0000 7fae0affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:25.696 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.717+0000 7fae0affd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:23:25.696 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.717+0000 7fae0affd640 1 -- 172.21.15.86:0/2511663031 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae0c146340 con 0x7fae0c0a9d70 2024-03-29T11:23:25.697 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.717+0000 7fae0affd640 10 monclient: _finish_auth 0 2024-03-29T11:23:25.697 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.717+0000 7fae0affd640 10 monclient: _check_auth_tickets 2024-03-29T11:23:25.697 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.717+0000 7fae0affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:25.697 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.717+0000 7fae0affd640 1 -- 172.21.15.86:0/2511663031 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fae04001a10 con 0x7fae0c0a9d70 2024-03-29T11:23:25.697 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.717+0000 7fae0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:25.697 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.717+0000 7fae0affd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:25.697 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.717+0000 7fae0affd640 10 monclient: dump: 2024-03-29T11:23:25.697 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:25.697 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:25.697 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:25.697 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:25.697 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:25.697 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:25.697 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:25.698 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:25.698 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:25.698 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:25.698 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.718+0000 7fae0affd640 1 -- 172.21.15.86:0/2511663031 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae04001be0 con 0x7fae0c0a9d70 2024-03-29T11:23:25.698 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.718+0000 7fae0affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:25.698 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.718+0000 7fae0affd640 1 -- 172.21.15.86:0/2511663031 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fae04002030 con 0x7fae0c0a9d70 2024-03-29T11:23:25.699 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.718+0000 7fae0affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:25.699 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.718+0000 7fae0affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:25.699 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.718+0000 7fae0affd640 10 monclient: dump: 2024-03-29T11:23:25.699 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:25.699 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:25.699 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:25.699 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:25.699 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:25.699 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:25.699 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:25.699 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:25.699 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:25.699 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:25.700 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.718+0000 7fae124d3640 5 monclient: authenticate success, global_id 4346 2024-03-29T11:23:25.700 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.718+0000 7fae124d3640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:25.700 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.718+0000 7fae124d3640 4 set_mon_vals no callback set 2024-03-29T11:23:25.700 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.719+0000 7fae124d3640 10 monclient: shutdown 2024-03-29T11:23:25.700 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.719+0000 7fae124d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:25.700 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.719+0000 7fae124d3640 1 -- 172.21.15.86:0/2511663031 >> v1:172.21.15.86:6790/0 conn(0x7fae0c0a9d70 legacy=0x7fae0c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:25.700 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.720+0000 7fae124d3640 1 -- 172.21.15.86:0/2511663031 shutdown_connections 2024-03-29T11:23:25.700 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.720+0000 7fae124d3640 1 -- 172.21.15.86:0/2511663031 >> 172.21.15.86:0/2511663031 conn(0x7fae0c066a00 msgr2=0x7fae0c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:25.702 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.723+0000 7fae124d3640 1 -- 172.21.15.86:0/2511663031 shutdown_connections 2024-03-29T11:23:25.703 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.724+0000 7fae124d3640 1 -- 172.21.15.86:0/2511663031 wait complete. 2024-03-29T11:23:25.703 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.724+0000 7fae124d3640 10 monclient: build_initial_monmap 2024-03-29T11:23:25.703 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.724+0000 7fae124d3640 10 monclient: monmap: 2024-03-29T11:23:25.704 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:23:25.704 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:25.704 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:23:25.704 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:23:25.704 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:25.704 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:25.704 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:23:25.704 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:23:25.704 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:23:25.704 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:25.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.725+0000 7fae124d3640 1 librados: starting msgr at 2024-03-29T11:23:25.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.725+0000 7fae124d3640 1 librados: starting objecter 2024-03-29T11:23:25.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.725+0000 7fae124d3640 1 Processor -- start 2024-03-29T11:23:25.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.725+0000 7fae124d3640 1 -- start start 2024-03-29T11:23:25.706 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.725+0000 7fae124d3640 1 librados: setting wanted keys 2024-03-29T11:23:25.706 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.725+0000 7fae124d3640 1 librados: calling monclient init 2024-03-29T11:23:25.706 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.725+0000 7fae124d3640 10 monclient: init 2024-03-29T11:23:25.706 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.726+0000 7fae124d3640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:25.706 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.726+0000 7fae124d3640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:23:25.708 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.729+0000 7fae124d3640 10 monclient(hunting): picked mon.noname-c con 0x7fae0c0a9d70 addr v1:172.21.15.86:6790/0 2024-03-29T11:23:25.719 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.729+0000 7fae124d3640 10 monclient(hunting): picked mon.noname-b con 0x7fae0c13f650 addr v1:172.21.15.196:6789/0 2024-03-29T11:23:25.719 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.729+0000 7fae124d3640 10 monclient(hunting): picked mon.noname-a con 0x7fae0c1457f0 addr v1:172.21.15.86:6789/0 2024-03-29T11:23:25.719 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.729+0000 7fae124d3640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae0c0702b0 con 0x7fae0c1457f0 2024-03-29T11:23:25.719 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.729+0000 7fae124d3640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae0c070590 con 0x7fae0c13f650 2024-03-29T11:23:25.719 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.729+0000 7fae124d3640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae0c13b1e0 con 0x7fae0c0a9d70 2024-03-29T11:23:25.719 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.729+0000 7fae124d3640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:25.719 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.729+0000 7fae124d3640 10 monclient(hunting): authenticate will time out at 1030.047913s 2024-03-29T11:23:25.719 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.732+0000 7fae10a70640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fae0c1457f0 0x7fae0c13aad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:59537/0 (socket says 172.21.15.86:59537) 2024-03-29T11:23:25.719 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.732+0000 7fae10a70640 1 -- 172.21.15.86:0/586044175 learned_addr learned my addr 172.21.15.86:0/586044175 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:23:25.719 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.732+0000 7fae0bfff640 1 --1- 172.21.15.86:0/586044175 >> v1:172.21.15.86:6790/0 conn(0x7fae0c0a9d70 0x7fae0c070a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:44969/0 (socket says 172.21.15.86:44969) 2024-03-29T11:23:25.719 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.733+0000 7fae08ff9640 10 client.?.objecter ms_handle_connect 0x7fae0c0a9d70 2024-03-29T11:23:25.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.733+0000 7fae08ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:25.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.733+0000 7fae08ff9640 10 client.?.objecter ms_handle_connect 0x7fae0c1457f0 2024-03-29T11:23:25.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.733+0000 7fae08ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:25.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.733+0000 7fae08ff9640 1 -- 172.21.15.86:0/586044175 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2955753548 0 0) 0x7fae0c0702b0 con 0x7fae0c1457f0 2024-03-29T11:23:25.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.733+0000 7fae08ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:23:25.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.733+0000 7fae08ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:25.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.733+0000 7fae08ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:25.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.733+0000 7fae08ff9640 10 monclient(hunting): my global_id is 4329 2024-03-29T11:23:25.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.733+0000 7fae08ff9640 1 -- 172.21.15.86:0/586044175 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadf8004eb0 con 0x7fae0c1457f0 2024-03-29T11:23:25.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.733+0000 7fae08ff9640 10 client.?.objecter ms_handle_connect 0x7fae0c13f650 2024-03-29T11:23:25.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.733+0000 7fae08ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:25.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.733+0000 7fae08ff9640 1 -- 172.21.15.86:0/586044175 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 385518263 0 0) 0x7fadf8004eb0 con 0x7fae0c1457f0 2024-03-29T11:23:25.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.733+0000 7fae08ff9640 1 -- 172.21.15.86:0/586044175 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fadf8005300 con 0x7fae0c1457f0 2024-03-29T11:23:25.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.733+0000 7fae08ff9640 1 -- 172.21.15.86:0/586044175 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4128553438 0 0) 0x7fae0c070590 con 0x7fae0c13f650 2024-03-29T11:23:25.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.733+0000 7fae08ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:23:25.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.733+0000 7fae08ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:25.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.733+0000 7fae08ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:25.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.733+0000 7fae08ff9640 10 monclient(hunting): my global_id is 4309 2024-03-29T11:23:25.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.733+0000 7fae08ff9640 1 -- 172.21.15.86:0/586044175 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fadf8005760 con 0x7fae0c13f650 2024-03-29T11:23:25.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.734+0000 7fae08ff9640 1 -- 172.21.15.86:0/586044175 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fae04001ef0 con 0x7fae0c1457f0 2024-03-29T11:23:25.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.734+0000 7fae08ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:25.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.734+0000 7fae08ff9640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:25.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.734+0000 7fae08ff9640 10 monclient(hunting): dump: 2024-03-29T11:23:25.721 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:25.721 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:25.721 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:25.721 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:25.721 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:25.722 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:25.722 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:25.722 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:25.722 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:25.722 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:25.722 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.734+0000 7fae08ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:25.722 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.734+0000 7fae08ff9640 1 -- 172.21.15.86:0/586044175 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2106549431 0 0) 0x7fae0c13b1e0 con 0x7fae0c0a9d70 2024-03-29T11:23:25.722 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.734+0000 7fae08ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:23:25.722 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.734+0000 7fae08ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:25.722 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.734+0000 7fae08ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:25.722 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.734+0000 7fae08ff9640 10 monclient(hunting): my global_id is 4349 2024-03-29T11:23:25.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.734+0000 7fae08ff9640 1 -- 172.21.15.86:0/586044175 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fae0c070590 con 0x7fae0c0a9d70 2024-03-29T11:23:25.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.734+0000 7fae08ff9640 1 -- 172.21.15.86:0/586044175 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1281103314 0 0) 0x7fadf8005760 con 0x7fae0c13f650 2024-03-29T11:23:25.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.734+0000 7fae08ff9640 1 -- 172.21.15.86:0/586044175 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fae0c13b1e0 con 0x7fae0c13f650 2024-03-29T11:23:25.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.734+0000 7fae08ff9640 1 -- 172.21.15.86:0/586044175 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fae00001870 con 0x7fae0c13f650 2024-03-29T11:23:25.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.734+0000 7fae08ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:25.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.734+0000 7fae08ff9640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:25.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.734+0000 7fae08ff9640 10 monclient(hunting): dump: 2024-03-29T11:23:25.723 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:25.723 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:25.723 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:25.723 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:25.723 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:25.723 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:25.724 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:25.724 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:25.724 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:25.724 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:25.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.734+0000 7fae08ff9640 1 -- 172.21.15.86:0/586044175 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2887013806 0 0) 0x7fae0c070590 con 0x7fae0c0a9d70 2024-03-29T11:23:25.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.734+0000 7fae08ff9640 1 -- 172.21.15.86:0/586044175 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fadf8005760 con 0x7fae0c0a9d70 2024-03-29T11:23:25.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.734+0000 7fae08ff9640 1 -- 172.21.15.86:0/586044175 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fadfc001710 con 0x7fae0c0a9d70 2024-03-29T11:23:25.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.734+0000 7fae08ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:25.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.734+0000 7fae08ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:25.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.734+0000 7fae08ff9640 10 monclient(hunting): dump: 2024-03-29T11:23:25.724 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:25.724 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:25.724 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:25.725 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:25.725 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:25.725 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:25.725 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:25.725 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:25.725 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:25.725 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:25.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.734+0000 7fae08ff9640 1 -- 172.21.15.86:0/586044175 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 237548783 0 0) 0x7fae0c13b1e0 con 0x7fae0c13f650 2024-03-29T11:23:25.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.735+0000 7fae08ff9640 1 -- 172.21.15.86:0/586044175 >> v1:172.21.15.86:6790/0 conn(0x7fae0c0a9d70 legacy=0x7fae0c070a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:25.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.735+0000 7fae08ff9640 1 -- 172.21.15.86:0/586044175 >> v1:172.21.15.86:6789/0 conn(0x7fae0c1457f0 legacy=0x7fae0c13aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:25.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.735+0000 7fae08ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:23:25.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.735+0000 7fae08ff9640 1 monclient: found mon.b 2024-03-29T11:23:25.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.735+0000 7fae08ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:25.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.735+0000 7fae08ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:23:25.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.735+0000 7fae08ff9640 1 -- 172.21.15.86:0/586044175 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae0c071fe0 con 0x7fae0c13f650 2024-03-29T11:23:25.726 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.735+0000 7fae08ff9640 10 monclient: _finish_auth 0 2024-03-29T11:23:25.726 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.735+0000 7fae08ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:23:25.726 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.735+0000 7fae08ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:25.726 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.735+0000 7fae124d3640 5 monclient: authenticate success, global_id 4309 2024-03-29T11:23:25.726 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.735+0000 7fae124d3640 10 monclient: _renew_subs 2024-03-29T11:23:25.726 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.735+0000 7fae124d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:23:25.726 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.735+0000 7fae124d3640 1 -- 172.21.15.86:0/586044175 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae0c071160 con 0x7fae0c13f650 2024-03-29T11:23:25.726 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.735+0000 7fae124d3640 10 client.4309.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:25.726 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.735+0000 7fae124d3640 10 monclient: _renew_subs 2024-03-29T11:23:25.726 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.735+0000 7fae124d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:23:25.726 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.735+0000 7fae124d3640 1 -- 172.21.15.86:0/586044175 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fae0c1ded00 con 0x7fae0c13f650 2024-03-29T11:23:25.726 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.735+0000 7fae124d3640 1 librados: init done 2024-03-29T11:23:25.726 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.735+0000 7fae08ff9640 1 -- 172.21.15.86:0/586044175 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fae00001d60 con 0x7fae0c13f650 2024-03-29T11:23:25.727 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.735+0000 7fae08ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:25.727 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.736+0000 7fae08ff9640 1 -- 172.21.15.86:0/586044175 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fae00003790 con 0x7fae0c13f650 2024-03-29T11:23:25.727 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.736+0000 7fae0a7fc640 4 set_mon_vals no callback set 2024-03-29T11:23:25.727 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.736+0000 7fae08ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:25.727 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.736+0000 7fae08ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:25.727 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.736+0000 7fae08ff9640 10 monclient: dump: 2024-03-29T11:23:25.727 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:25.727 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:25.727 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:25.727 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:25.727 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:25.727 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:25.727 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:25.727 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:25.727 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:25.728 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:25.728 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.736+0000 7fae124d3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:25.728 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.737+0000 7fae124d3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:25.728 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.737+0000 7fae124d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:23:25.728 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.737+0000 7fae124d3640 1 -- 172.21.15.86:0/586044175 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadec005080 con 0x7fae0c13f650 2024-03-29T11:23:25.728 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.737+0000 7fae08ff9640 1 -- 172.21.15.86:0/586044175 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7fae000038f0 con 0x7fae0c13f650 2024-03-29T11:23:25.728 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.738+0000 7fae08ff9640 1 -- 172.21.15.86:0/586044175 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7fae000837a0 con 0x7fae0c13f650 2024-03-29T11:23:25.728 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.738+0000 7fae08ff9640 10 client.4309.objecter ms_dispatch 0x7fae0c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:25.728 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.738+0000 7fae08ff9640 3 client.4309.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:25.728 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.738+0000 7fae08ff9640 3 client.4309.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:25.728 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.738+0000 7fae08ff9640 20 client.4309.objecter dump_active .. 0 homeless 2024-03-29T11:23:25.729 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.740+0000 7fae0bfff640 10 monclient: get_auth_request con 0x7fadf806e600 auth_method 0 2024-03-29T11:23:25.729 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.741+0000 7fae08ff9640 10 client.4309.objecter ms_handle_connect 0x7fadf806e600 2024-03-29T11:23:25.729 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.746+0000 7fae08ff9640 1 -- 172.21.15.86:0/586044175 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fae0004f0a0 con 0x7fae0c13f650 2024-03-29T11:23:25.729 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.746+0000 7fae08ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:25.729 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:25.746+0000 7fae08ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:26.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:26.072+0000 7fae124d3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:26.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:26.072+0000 7fae124d3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:26.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:26.072+0000 7fae124d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:23:26.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:26.072+0000 7fae124d3640 1 -- 172.21.15.86:0/586044175 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fadec005330 con 0x7fae0c13f650 2024-03-29T11:23:26.054 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:26.075+0000 7fae08ff9640 1 -- 172.21.15.86:0/586044175 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7fae00050fb0 con 0x7fae0c13f650 2024-03-29T11:23:26.054 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:26.075+0000 7fae08ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:26.054 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:26.075+0000 7fae08ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:26.054 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:23:26.054 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:26.057 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:26.078+0000 7fae124d3640 10 librados: watch_flush enter 2024-03-29T11:23:26.058 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:26.078+0000 7fae124d3640 10 librados: watch_flush exit 2024-03-29T11:23:26.058 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:26.078+0000 7fae124d3640 20 client.4309.objecter shutdown clearing up homeless session... 2024-03-29T11:23:26.058 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:26.078+0000 7fae124d3640 10 client.4309.objecter successfully canceled tick 2024-03-29T11:23:26.058 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:26.079+0000 7fae124d3640 1 -- 172.21.15.86:0/586044175 >> v1:172.21.15.196:6816/588110318 conn(0x7fadf806e600 legacy=0x7fadf8070ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:26.058 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:26.079+0000 7fae124d3640 10 monclient: shutdown 2024-03-29T11:23:26.058 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:26.079+0000 7fae124d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:26.058 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:26.079+0000 7fae124d3640 1 -- 172.21.15.86:0/586044175 >> v1:172.21.15.196:6789/0 conn(0x7fae0c13f650 legacy=0x7fae0c13a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:26.058 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:26.079+0000 7fae124d3640 1 -- 172.21.15.86:0/586044175 shutdown_connections 2024-03-29T11:23:26.058 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:26.079+0000 7fae124d3640 1 -- 172.21.15.86:0/586044175 >> 172.21.15.86:0/586044175 conn(0x7fae0c066a00 msgr2=0x7fae0c0b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:26.059 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:26.079+0000 7fae124d3640 1 -- 172.21.15.86:0/586044175 shutdown_connections 2024-03-29T11:23:26.059 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:26.079+0000 7fae124d3640 1 -- 172.21.15.86:0/586044175 wait complete. 2024-03-29T11:23:26.059 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:26.080+0000 7fae124d3640 1 librados: shutdown 2024-03-29T11:23:26.075 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:32.882 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:33.037 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.058+0000 7f6a6759e640 10 monclient: get_monmap_and_config 2024-03-29T11:23:33.037 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.058+0000 7f6a6759e640 10 monclient: build_initial_monmap 2024-03-29T11:23:33.037 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.058+0000 7f6a6759e640 10 monclient: monmap: 2024-03-29T11:23:33.037 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:23:33.038 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:33.038 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:23:33.059148+0000 2024-03-29T11:23:33.038 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:23:33.059148+0000 2024-03-29T11:23:33.038 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:33.038 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:33.038 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:23:33.038 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:23:33.038 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:23:33.038 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:33.040 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.060+0000 7f6a6759e640 1 Processor -- start 2024-03-29T11:23:33.040 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.060+0000 7f6a6759e640 1 -- start start 2024-03-29T11:23:33.040 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.060+0000 7f6a6759e640 10 monclient: init 2024-03-29T11:23:33.041 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.061+0000 7f6a6759e640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:33.041 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.061+0000 7f6a6759e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:33.041 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.061+0000 7f6a6759e640 10 monclient(hunting): picked mon.noname-c con 0x7f6a680aad20 addr v1:172.21.15.196:6789/0 2024-03-29T11:23:33.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.061+0000 7f6a6759e640 10 monclient(hunting): picked mon.noname-a con 0x7f6a6806d8c0 addr v1:172.21.15.86:6789/0 2024-03-29T11:23:33.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.061+0000 7f6a6759e640 10 monclient(hunting): picked mon.noname-b con 0x7f6a680afc40 addr v1:172.21.15.86:6790/0 2024-03-29T11:23:33.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.061+0000 7f6a6759e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a680af4d0 con 0x7f6a6806d8c0 2024-03-29T11:23:33.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.061+0000 7f6a6759e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a680af7b0 con 0x7f6a680aad20 2024-03-29T11:23:33.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.061+0000 7f6a6759e640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a680acab0 con 0x7f6a680afc40 2024-03-29T11:23:33.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.061+0000 7f6a6759e640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:33.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.061+0000 7f6a6759e640 10 monclient(hunting): authenticate will time out at 1037.379758s 2024-03-29T11:23:33.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.065+0000 7f6a66d9d640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f6a680afc40 0x7f6a68141a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:54613/0 (socket says 172.21.15.86:54613) 2024-03-29T11:23:33.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.065+0000 7f6a66d9d640 1 -- 172.21.15.86:0/822626474 learned_addr learned my addr 172.21.15.86:0/822626474 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:23:33.047 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.066+0000 7f6a6559a640 1 -- 172.21.15.86:0/822626474 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1784146677 0 0) 0x7f6a680af4d0 con 0x7f6a6806d8c0 2024-03-29T11:23:33.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.066+0000 7f6a6559a640 10 monclient(hunting): _negotiate 2024-03-29T11:23:33.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.066+0000 7f6a6559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:33.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.066+0000 7f6a6559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:33.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.066+0000 7f6a6559a640 10 monclient(hunting): my global_id is 4332 2024-03-29T11:23:33.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.066+0000 7f6a6559a640 1 -- 172.21.15.86:0/822626474 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a50003520 con 0x7f6a6806d8c0 2024-03-29T11:23:33.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.066+0000 7f6a6559a640 1 -- 172.21.15.86:0/822626474 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1949898454 0 0) 0x7f6a50003520 con 0x7f6a6806d8c0 2024-03-29T11:23:33.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.066+0000 7f6a6559a640 1 -- 172.21.15.86:0/822626474 >> v1:172.21.15.86:6790/0 conn(0x7f6a680afc40 legacy=0x7f6a68141a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:33.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.066+0000 7f6a6559a640 1 -- 172.21.15.86:0/822626474 >> v1:172.21.15.196:6789/0 conn(0x7f6a680aad20 legacy=0x7f6a680ac3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:23:33.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.066+0000 7f6a6559a640 10 monclient: _finish_hunting 0 2024-03-29T11:23:33.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.066+0000 7f6a6559a640 1 monclient: found mon.noname-a 2024-03-29T11:23:33.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.066+0000 7f6a6559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:33.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.066+0000 7f6a6559a640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.86:6789/0 2024-03-29T11:23:33.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.066+0000 7f6a6559a640 1 -- 172.21.15.86:0/822626474 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a680aba30 con 0x7f6a6806d8c0 2024-03-29T11:23:33.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.066+0000 7f6a6559a640 10 monclient: _finish_auth 0 2024-03-29T11:23:33.049 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.066+0000 7f6a6559a640 10 monclient: _check_auth_tickets 2024-03-29T11:23:33.049 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.066+0000 7f6a6559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:33.049 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.066+0000 7f6a6559a640 1 -- 172.21.15.86:0/822626474 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f6a58001690 con 0x7f6a6806d8c0 2024-03-29T11:23:33.049 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.067+0000 7f6a6559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:33.049 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.067+0000 7f6a6559a640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:33.049 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.067+0000 7f6a6559a640 10 monclient: dump: 2024-03-29T11:23:33.049 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:33.049 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:33.049 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:33.049 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:33.049 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:33.049 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:33.049 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:33.049 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:33.049 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:33.049 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:33.049 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.067+0000 7f6a6559a640 1 -- 172.21.15.86:0/822626474 <== mon.0 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a580017f0 con 0x7f6a6806d8c0 2024-03-29T11:23:33.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.067+0000 7f6a6559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:33.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.067+0000 7f6a6559a640 1 -- 172.21.15.86:0/822626474 <== mon.0 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f6a58001c00 con 0x7f6a6806d8c0 2024-03-29T11:23:33.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.067+0000 7f6a6559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:33.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.067+0000 7f6a6559a640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:33.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.067+0000 7f6a6559a640 10 monclient: dump: 2024-03-29T11:23:33.050 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:33.050 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:33.050 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:33.050 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:33.050 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:33.050 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:33.050 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:33.050 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:33.050 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:33.050 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:33.051 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.068+0000 7f6a6759e640 5 monclient: authenticate success, global_id 4332 2024-03-29T11:23:33.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.068+0000 7f6a6759e640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:33.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.069+0000 7f6a6759e640 4 set_mon_vals no callback set 2024-03-29T11:23:33.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.069+0000 7f6a6759e640 10 monclient: shutdown 2024-03-29T11:23:33.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.069+0000 7f6a6759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:33.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.069+0000 7f6a6759e640 1 -- 172.21.15.86:0/822626474 >> v1:172.21.15.86:6789/0 conn(0x7f6a6806d8c0 legacy=0x7f6a680a94e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:33.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.072+0000 7f6a6759e640 1 -- 172.21.15.86:0/822626474 shutdown_connections 2024-03-29T11:23:33.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.072+0000 7f6a6759e640 1 -- 172.21.15.86:0/822626474 >> 172.21.15.86:0/822626474 conn(0x7f6a680673e0 msgr2=0x7f6a680b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:33.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.072+0000 7f6a6759e640 1 -- 172.21.15.86:0/822626474 shutdown_connections 2024-03-29T11:23:33.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.072+0000 7f6a6759e640 1 -- 172.21.15.86:0/822626474 wait complete. 2024-03-29T11:23:33.057 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.073+0000 7f6a6759e640 10 monclient: build_initial_monmap 2024-03-29T11:23:33.057 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.074+0000 7f6a6759e640 10 monclient: monmap: 2024-03-29T11:23:33.057 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:23:33.058 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:33.058 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:23:33.058 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:23:33.058 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:33.058 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:33.058 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:23:33.058 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:23:33.058 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:23:33.058 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:33.058 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.074+0000 7f6a6759e640 1 librados: starting msgr at 2024-03-29T11:23:33.058 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.074+0000 7f6a6759e640 1 librados: starting objecter 2024-03-29T11:23:33.058 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.074+0000 7f6a6759e640 1 Processor -- start 2024-03-29T11:23:33.058 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.074+0000 7f6a6759e640 1 -- start start 2024-03-29T11:23:33.058 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.074+0000 7f6a6759e640 1 librados: setting wanted keys 2024-03-29T11:23:33.058 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.074+0000 7f6a6759e640 1 librados: calling monclient init 2024-03-29T11:23:33.059 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.074+0000 7f6a6759e640 10 monclient: init 2024-03-29T11:23:33.059 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.075+0000 7f6a6759e640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:33.059 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.075+0000 7f6a6759e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:23:33.059 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.075+0000 7f6a6759e640 10 monclient(hunting): picked mon.noname-a con 0x7f6a6806d8c0 addr v1:172.21.15.86:6789/0 2024-03-29T11:23:33.059 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.075+0000 7f6a6759e640 10 monclient(hunting): picked mon.noname-b con 0x7f6a680aad20 addr v1:172.21.15.196:6789/0 2024-03-29T11:23:33.059 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.075+0000 7f6a6759e640 10 monclient(hunting): picked mon.noname-c con 0x7f6a680afc40 addr v1:172.21.15.86:6790/0 2024-03-29T11:23:33.059 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.075+0000 7f6a6759e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a6812b360 con 0x7f6a6806d8c0 2024-03-29T11:23:33.059 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.075+0000 7f6a6759e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a6812f650 con 0x7f6a680aad20 2024-03-29T11:23:33.059 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.075+0000 7f6a6759e640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a6812f930 con 0x7f6a680afc40 2024-03-29T11:23:33.059 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.075+0000 7f6a6759e640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:33.059 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.075+0000 7f6a6759e640 10 monclient(hunting): authenticate will time out at 1037.393369s 2024-03-29T11:23:33.060 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.080+0000 7f6a6659c640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f6a6806d8c0 0x7f6a6812e0c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:50467/0 (socket says 172.21.15.86:50467) 2024-03-29T11:23:33.060 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.080+0000 7f6a6659c640 1 -- 172.21.15.86:0/4259716557 learned_addr learned my addr 172.21.15.86:0/4259716557 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:23:33.060 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.081+0000 7f6a56ffd640 10 client.?.objecter ms_handle_connect 0x7f6a6806d8c0 2024-03-29T11:23:33.061 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.082+0000 7f6a56ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:33.061 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.082+0000 7f6a56ffd640 10 client.?.objecter ms_handle_connect 0x7f6a680aad20 2024-03-29T11:23:33.061 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.082+0000 7f6a56ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:33.064 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.083+0000 7f6a56ffd640 1 -- 172.21.15.86:0/4259716557 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3046377522 0 0) 0x7f6a6812f650 con 0x7f6a680aad20 2024-03-29T11:23:33.065 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.083+0000 7f6a56ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:33.065 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.083+0000 7f6a56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:33.065 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.083+0000 7f6a56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:33.065 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.083+0000 7f6a56ffd640 10 monclient(hunting): my global_id is 4312 2024-03-29T11:23:33.065 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.083+0000 7f6a56ffd640 1 -- 172.21.15.86:0/4259716557 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a48004eb0 con 0x7f6a680aad20 2024-03-29T11:23:33.065 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.083+0000 7f6a56ffd640 1 -- 172.21.15.86:0/4259716557 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4289187828 0 0) 0x7f6a6812b360 con 0x7f6a6806d8c0 2024-03-29T11:23:33.065 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.084+0000 7f6a56ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:33.065 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.084+0000 7f6a56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:33.065 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.084+0000 7f6a56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:33.065 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.084+0000 7f6a56ffd640 10 monclient(hunting): my global_id is 4335 2024-03-29T11:23:33.065 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.084+0000 7f6a56ffd640 1 -- 172.21.15.86:0/4259716557 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a6812f650 con 0x7f6a6806d8c0 2024-03-29T11:23:33.065 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.084+0000 7f6a56ffd640 10 client.?.objecter ms_handle_connect 0x7f6a680afc40 2024-03-29T11:23:33.065 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.084+0000 7f6a56ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:33.065 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.084+0000 7f6a56ffd640 1 -- 172.21.15.86:0/4259716557 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3247580873 0 0) 0x7f6a48004eb0 con 0x7f6a680aad20 2024-03-29T11:23:33.065 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.084+0000 7f6a56ffd640 1 -- 172.21.15.86:0/4259716557 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a6812b360 con 0x7f6a680aad20 2024-03-29T11:23:33.066 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.084+0000 7f6a56ffd640 1 -- 172.21.15.86:0/4259716557 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2142976730 0 0) 0x7f6a6812f930 con 0x7f6a680afc40 2024-03-29T11:23:33.066 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.085+0000 7f6a56ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:33.066 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.085+0000 7f6a56ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:33.066 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.085+0000 7f6a56ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:33.066 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.085+0000 7f6a56ffd640 10 monclient(hunting): my global_id is 4355 2024-03-29T11:23:33.066 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.085+0000 7f6a56ffd640 1 -- 172.21.15.86:0/4259716557 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a48004eb0 con 0x7f6a680afc40 2024-03-29T11:23:33.066 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.085+0000 7f6a56ffd640 1 -- 172.21.15.86:0/4259716557 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f6a58001bc0 con 0x7f6a680aad20 2024-03-29T11:23:33.066 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.085+0000 7f6a56ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:33.066 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.085+0000 7f6a56ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:33.066 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.085+0000 7f6a56ffd640 10 monclient(hunting): dump: 2024-03-29T11:23:33.066 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:33.066 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:33.066 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:33.066 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:33.067 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:33.067 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:33.067 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:33.067 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:33.067 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:33.067 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:33.067 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.085+0000 7f6a56ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:33.067 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.085+0000 7f6a56ffd640 1 -- 172.21.15.86:0/4259716557 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 874545962 0 0) 0x7f6a48004eb0 con 0x7f6a680afc40 2024-03-29T11:23:33.067 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.085+0000 7f6a56ffd640 1 -- 172.21.15.86:0/4259716557 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a6812f930 con 0x7f6a680afc40 2024-03-29T11:23:33.067 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.085+0000 7f6a56ffd640 1 -- 172.21.15.86:0/4259716557 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1087150568 0 0) 0x7f6a6812b360 con 0x7f6a680aad20 2024-03-29T11:23:33.067 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.085+0000 7f6a56ffd640 1 -- 172.21.15.86:0/4259716557 >> v1:172.21.15.86:6790/0 conn(0x7f6a680afc40 legacy=0x7f6a6812fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:33.067 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.085+0000 7f6a56ffd640 1 -- 172.21.15.86:0/4259716557 >> v1:172.21.15.86:6789/0 conn(0x7f6a6806d8c0 legacy=0x7f6a6812e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:33.067 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.085+0000 7f6a56ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:23:33.067 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.085+0000 7f6a56ffd640 1 monclient: found mon.b 2024-03-29T11:23:33.067 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.085+0000 7f6a56ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:33.067 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.085+0000 7f6a56ffd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:23:33.068 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.085+0000 7f6a56ffd640 1 -- 172.21.15.86:0/4259716557 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a6812b010 con 0x7f6a680aad20 2024-03-29T11:23:33.068 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.085+0000 7f6a56ffd640 10 monclient: _finish_auth 0 2024-03-29T11:23:33.068 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.085+0000 7f6a56ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:23:33.068 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.085+0000 7f6a56ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:33.068 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.085+0000 7f6a6759e640 5 monclient: authenticate success, global_id 4312 2024-03-29T11:23:33.068 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.085+0000 7f6a6759e640 10 monclient: _renew_subs 2024-03-29T11:23:33.068 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.085+0000 7f6a6759e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:23:33.068 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.085+0000 7f6a6759e640 1 -- 172.21.15.86:0/4259716557 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a6812d2d0 con 0x7f6a680aad20 2024-03-29T11:23:33.068 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.085+0000 7f6a6759e640 10 client.4312.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:33.068 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.085+0000 7f6a6759e640 10 monclient: _renew_subs 2024-03-29T11:23:33.068 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.085+0000 7f6a6759e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:23:33.068 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.085+0000 7f6a6759e640 1 -- 172.21.15.86:0/4259716557 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a6812d570 con 0x7f6a680aad20 2024-03-29T11:23:33.068 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.085+0000 7f6a6759e640 1 librados: init done 2024-03-29T11:23:33.069 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.086+0000 7f6a56ffd640 1 -- 172.21.15.86:0/4259716557 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a58003910 con 0x7f6a680aad20 2024-03-29T11:23:33.069 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.087+0000 7f6a6759e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:33.069 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.087+0000 7f6a6759e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:33.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.087+0000 7f6a6759e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:23:33.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.087+0000 7f6a6759e640 1 -- 172.21.15.86:0/4259716557 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a6804f730 con 0x7f6a680aad20 2024-03-29T11:23:33.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.087+0000 7f6a56ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:33.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.087+0000 7f6a56ffd640 1 -- 172.21.15.86:0/4259716557 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f6a58003c60 con 0x7f6a680aad20 2024-03-29T11:23:33.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.087+0000 7f6a56ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:33.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.088+0000 7f6a56ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:33.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.088+0000 7f6a56ffd640 10 monclient: dump: 2024-03-29T11:23:33.070 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:33.070 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:33.070 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:33.070 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:33.070 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:33.070 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:33.071 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:33.071 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:33.071 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:33.071 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:33.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.088+0000 7f6a56ffd640 1 -- 172.21.15.86:0/4259716557 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f6a58003e80 con 0x7f6a680aad20 2024-03-29T11:23:33.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.088+0000 7f6a64d99640 4 set_mon_vals no callback set 2024-03-29T11:23:33.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.089+0000 7f6a56ffd640 1 -- 172.21.15.86:0/4259716557 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f6a58083970 con 0x7f6a680aad20 2024-03-29T11:23:33.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.089+0000 7f6a56ffd640 10 client.4312.objecter ms_dispatch 0x7f6a68000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:33.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.089+0000 7f6a56ffd640 3 client.4312.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:33.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.089+0000 7f6a56ffd640 3 client.4312.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:33.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.089+0000 7f6a56ffd640 20 client.4312.objecter dump_active .. 0 homeless 2024-03-29T11:23:33.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.090+0000 7f6a6659c640 10 monclient: get_auth_request con 0x7f6a4806df40 auth_method 0 2024-03-29T11:23:33.072 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.091+0000 7f6a56ffd640 10 client.4312.objecter ms_handle_connect 0x7f6a4806df40 2024-03-29T11:23:33.075 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.095+0000 7f6a56ffd640 1 -- 172.21.15.86:0/4259716557 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6a5804f2d0 con 0x7f6a680aad20 2024-03-29T11:23:33.075 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.096+0000 7f6a56ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:33.075 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.096+0000 7f6a56ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:33.408 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.427+0000 7f6a6759e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:33.408 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.427+0000 7f6a6759e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:33.408 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.427+0000 7f6a6759e640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:23:33.408 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.427+0000 7f6a6759e640 1 -- 172.21.15.86:0/4259716557 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a68074e10 con 0x7f6a680aad20 2024-03-29T11:23:33.408 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.429+0000 7f6a56ffd640 1 -- 172.21.15.86:0/4259716557 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f6a58051200 con 0x7f6a680aad20 2024-03-29T11:23:33.408 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.429+0000 7f6a56ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:33.408 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.429+0000 7f6a56ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:33.410 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:23:33.411 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:33.412 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.432+0000 7f6a6759e640 10 librados: watch_flush enter 2024-03-29T11:23:33.412 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.432+0000 7f6a6759e640 10 librados: watch_flush exit 2024-03-29T11:23:33.412 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.432+0000 7f6a6759e640 20 client.4312.objecter shutdown clearing up homeless session... 2024-03-29T11:23:33.412 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.432+0000 7f6a6759e640 10 client.4312.objecter successfully canceled tick 2024-03-29T11:23:33.413 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.434+0000 7f6a6759e640 1 -- 172.21.15.86:0/4259716557 >> v1:172.21.15.196:6816/588110318 conn(0x7f6a4806df40 legacy=0x7f6a48070400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:33.413 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.434+0000 7f6a6759e640 10 monclient: shutdown 2024-03-29T11:23:33.413 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.434+0000 7f6a6759e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:33.413 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.434+0000 7f6a6759e640 1 -- 172.21.15.86:0/4259716557 >> v1:172.21.15.196:6789/0 conn(0x7f6a680aad20 legacy=0x7f6a68071260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:33.417 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.437+0000 7f6a6759e640 1 -- 172.21.15.86:0/4259716557 shutdown_connections 2024-03-29T11:23:33.417 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.437+0000 7f6a6759e640 1 -- 172.21.15.86:0/4259716557 >> 172.21.15.86:0/4259716557 conn(0x7f6a680673e0 msgr2=0x7f6a680ac3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:33.418 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.439+0000 7f6a6759e640 1 -- 172.21.15.86:0/4259716557 shutdown_connections 2024-03-29T11:23:33.418 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.439+0000 7f6a6759e640 1 -- 172.21.15.86:0/4259716557 wait complete. 2024-03-29T11:23:33.419 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:33.440+0000 7f6a6759e640 1 librados: shutdown 2024-03-29T11:23:33.435 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:40.242 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:40.391 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.411+0000 7fa8a6bcf640 10 monclient: get_monmap_and_config 2024-03-29T11:23:40.391 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.411+0000 7fa8a6bcf640 10 monclient: build_initial_monmap 2024-03-29T11:23:40.392 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.411+0000 7fa8a6bcf640 10 monclient: monmap: 2024-03-29T11:23:40.392 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:23:40.392 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:40.392 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:23:40.412841+0000 2024-03-29T11:23:40.392 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:23:40.412841+0000 2024-03-29T11:23:40.392 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:40.392 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:40.392 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:23:40.392 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:23:40.392 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:23:40.392 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:40.393 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.412+0000 7fa8a6bcf640 1 Processor -- start 2024-03-29T11:23:40.393 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.412+0000 7fa8a6bcf640 1 -- start start 2024-03-29T11:23:40.393 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.412+0000 7fa8a6bcf640 10 monclient: init 2024-03-29T11:23:40.395 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.413+0000 7fa8a6bcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:40.395 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.413+0000 7fa8a6bcf640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:23:40.395 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.414+0000 7fa8a6bcf640 10 monclient(hunting): picked mon.noname-c con 0x7fa8a0071de0 addr v1:172.21.15.196:6789/0 2024-03-29T11:23:40.395 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.414+0000 7fa8a6bcf640 10 monclient(hunting): picked mon.noname-a con 0x7fa8a00721c0 addr v1:172.21.15.86:6789/0 2024-03-29T11:23:40.395 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.414+0000 7fa8a6bcf640 10 monclient(hunting): picked mon.noname-b con 0x7fa8a0145ae0 addr v1:172.21.15.86:6790/0 2024-03-29T11:23:40.395 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.414+0000 7fa8a6bcf640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a0145650 con 0x7fa8a00721c0 2024-03-29T11:23:40.395 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.414+0000 7fa8a6bcf640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a00ae260 con 0x7fa8a0071de0 2024-03-29T11:23:40.396 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.414+0000 7fa8a6bcf640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a01420d0 con 0x7fa8a0145ae0 2024-03-29T11:23:40.396 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.414+0000 7fa8a6bcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:40.396 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.414+0000 7fa8a6bcf640 10 monclient(hunting): authenticate will time out at 1044.732550s 2024-03-29T11:23:40.398 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.419+0000 7fa8a496b640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fa8a0071de0 0x7fa8a00aed80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:36085/0 (socket says 172.21.15.86:36085) 2024-03-29T11:23:40.398 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.419+0000 7fa8a496b640 1 -- 172.21.15.86:0/774178636 learned_addr learned my addr 172.21.15.86:0/774178636 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:23:40.402 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.420+0000 7fa89f7fe640 1 -- 172.21.15.86:0/774178636 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2274510233 0 0) 0x7fa8a00ae260 con 0x7fa8a0071de0 2024-03-29T11:23:40.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.420+0000 7fa89f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:40.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.420+0000 7fa89f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:40.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.420+0000 7fa89f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:40.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.420+0000 7fa89f7fe640 10 monclient(hunting): my global_id is 4315 2024-03-29T11:23:40.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.420+0000 7fa89f7fe640 1 -- 172.21.15.86:0/774178636 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa890003520 con 0x7fa8a0071de0 2024-03-29T11:23:40.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.420+0000 7fa89f7fe640 1 -- 172.21.15.86:0/774178636 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1965742454 0 0) 0x7fa8a0145650 con 0x7fa8a00721c0 2024-03-29T11:23:40.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.420+0000 7fa89f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:40.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.420+0000 7fa89f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:40.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.420+0000 7fa89f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:40.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.420+0000 7fa89f7fe640 10 monclient(hunting): my global_id is 4338 2024-03-29T11:23:40.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.420+0000 7fa89f7fe640 1 -- 172.21.15.86:0/774178636 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8a00ae260 con 0x7fa8a00721c0 2024-03-29T11:23:40.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.421+0000 7fa89f7fe640 1 -- 172.21.15.86:0/774178636 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2036892190 0 0) 0x7fa8a01420d0 con 0x7fa8a0145ae0 2024-03-29T11:23:40.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.421+0000 7fa89f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:40.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.421+0000 7fa89f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:40.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.421+0000 7fa89f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:40.404 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.421+0000 7fa89f7fe640 10 monclient(hunting): my global_id is 4358 2024-03-29T11:23:40.404 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.421+0000 7fa89f7fe640 1 -- 172.21.15.86:0/774178636 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8900039d0 con 0x7fa8a0145ae0 2024-03-29T11:23:40.404 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.421+0000 7fa89f7fe640 1 -- 172.21.15.86:0/774178636 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4274702310 0 0) 0x7fa8900039d0 con 0x7fa8a0145ae0 2024-03-29T11:23:40.404 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.421+0000 7fa89f7fe640 1 -- 172.21.15.86:0/774178636 >> v1:172.21.15.196:6789/0 conn(0x7fa8a0071de0 legacy=0x7fa8a00aed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:40.404 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.421+0000 7fa89f7fe640 1 -- 172.21.15.86:0/774178636 >> v1:172.21.15.86:6789/0 conn(0x7fa8a00721c0 legacy=0x7fa8a013f400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:40.404 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.421+0000 7fa89f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:23:40.404 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.421+0000 7fa89f7fe640 1 monclient: found mon.noname-b 2024-03-29T11:23:40.404 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.421+0000 7fa89f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:40.404 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.421+0000 7fa89f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:23:40.404 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.421+0000 7fa89f7fe640 1 -- 172.21.15.86:0/774178636 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a00b0590 con 0x7fa8a0145ae0 2024-03-29T11:23:40.404 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.421+0000 7fa89f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:23:40.404 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.421+0000 7fa89f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:23:40.404 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.421+0000 7fa89f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:40.404 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.421+0000 7fa89f7fe640 1 -- 172.21.15.86:0/774178636 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fa898001930 con 0x7fa8a0145ae0 2024-03-29T11:23:40.404 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.421+0000 7fa89f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:40.405 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.421+0000 7fa89f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:40.405 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.421+0000 7fa89f7fe640 10 monclient: dump: 2024-03-29T11:23:40.405 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:40.405 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:40.405 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:40.405 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:40.405 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:40.405 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:40.405 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:40.405 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:40.405 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:40.405 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:40.406 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.424+0000 7fa8a6bcf640 5 monclient: authenticate success, global_id 4358 2024-03-29T11:23:40.406 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.424+0000 7fa8a6bcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:40.407 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.424+0000 7fa89f7fe640 1 -- 172.21.15.86:0/774178636 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa898001a90 con 0x7fa8a0145ae0 2024-03-29T11:23:40.407 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.424+0000 7fa89f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:40.407 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.424+0000 7fa89f7fe640 1 -- 172.21.15.86:0/774178636 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fa898001e20 con 0x7fa8a0145ae0 2024-03-29T11:23:40.407 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.424+0000 7fa89f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:40.407 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.424+0000 7fa89f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:40.407 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.424+0000 7fa89f7fe640 10 monclient: dump: 2024-03-29T11:23:40.407 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:40.407 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:40.407 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:40.407 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:40.407 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:40.407 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:40.407 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:40.407 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:40.407 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:40.407 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:40.408 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.424+0000 7fa8a6bcf640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:40.408 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.424+0000 7fa8a6bcf640 4 set_mon_vals no callback set 2024-03-29T11:23:40.408 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.425+0000 7fa8a6bcf640 10 monclient: shutdown 2024-03-29T11:23:40.408 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.425+0000 7fa8a6bcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:40.408 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.425+0000 7fa8a6bcf640 1 -- 172.21.15.86:0/774178636 >> v1:172.21.15.86:6790/0 conn(0x7fa8a0145ae0 legacy=0x7fa8a0145f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:40.408 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.426+0000 7fa8a6bcf640 1 -- 172.21.15.86:0/774178636 shutdown_connections 2024-03-29T11:23:40.408 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.426+0000 7fa8a6bcf640 1 -- 172.21.15.86:0/774178636 >> 172.21.15.86:0/774178636 conn(0x7fa8a006b660 msgr2=0x7fa8a006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:40.408 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.426+0000 7fa8a6bcf640 1 -- 172.21.15.86:0/774178636 shutdown_connections 2024-03-29T11:23:40.408 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.426+0000 7fa8a6bcf640 1 -- 172.21.15.86:0/774178636 wait complete. 2024-03-29T11:23:40.409 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.427+0000 7fa8a6bcf640 10 monclient: build_initial_monmap 2024-03-29T11:23:40.409 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.427+0000 7fa8a6bcf640 10 monclient: monmap: 2024-03-29T11:23:40.409 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:23:40.409 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:40.409 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:23:40.409 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:23:40.409 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:40.409 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:40.409 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:23:40.409 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:23:40.409 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:23:40.409 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:40.409 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.427+0000 7fa8a6bcf640 1 librados: starting msgr at 2024-03-29T11:23:40.409 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.427+0000 7fa8a6bcf640 1 librados: starting objecter 2024-03-29T11:23:40.409 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.428+0000 7fa8a6bcf640 1 Processor -- start 2024-03-29T11:23:40.410 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.428+0000 7fa8a6bcf640 1 -- start start 2024-03-29T11:23:40.410 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.428+0000 7fa8a6bcf640 1 librados: setting wanted keys 2024-03-29T11:23:40.410 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.428+0000 7fa8a6bcf640 1 librados: calling monclient init 2024-03-29T11:23:40.410 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.428+0000 7fa8a6bcf640 10 monclient: init 2024-03-29T11:23:40.410 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.429+0000 7fa8a6bcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:40.410 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.429+0000 7fa8a6bcf640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:23:40.412 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.430+0000 7fa8a6bcf640 10 monclient(hunting): picked mon.noname-a con 0x7fa8a0071de0 addr v1:172.21.15.86:6789/0 2024-03-29T11:23:40.412 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.432+0000 7fa8a6bcf640 10 monclient(hunting): picked mon.noname-b con 0x7fa8a0145ae0 addr v1:172.21.15.196:6789/0 2024-03-29T11:23:40.414 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.434+0000 7fa8a6bcf640 10 monclient(hunting): picked mon.noname-c con 0x7fa8a01378d0 addr v1:172.21.15.86:6790/0 2024-03-29T11:23:40.414 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.434+0000 7fa8a6bcf640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a013aec0 con 0x7fa8a0071de0 2024-03-29T11:23:40.414 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.434+0000 7fa8a6bcf640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a013a430 con 0x7fa8a0145ae0 2024-03-29T11:23:40.414 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.434+0000 7fa8a6bcf640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8a013b8b0 con 0x7fa8a01378d0 2024-03-29T11:23:40.414 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.434+0000 7fa8a6bcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:40.415 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.434+0000 7fa8a6bcf640 10 monclient(hunting): authenticate will time out at 1044.752539s 2024-03-29T11:23:40.415 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.434+0000 7fa8a496b640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fa8a0071de0 0x7fa8a013c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:45281/0 (socket says 172.21.15.86:45281) 2024-03-29T11:23:40.415 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.434+0000 7fa8a496b640 1 -- 172.21.15.86:0/386720371 learned_addr learned my addr 172.21.15.86:0/386720371 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:23:40.417 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.437+0000 7fa89d7fa640 10 client.?.objecter ms_handle_connect 0x7fa8a0071de0 2024-03-29T11:23:40.417 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.437+0000 7fa89d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:40.417 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.437+0000 7fa89d7fa640 1 -- 172.21.15.86:0/386720371 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3350785104 0 0) 0x7fa8a013aec0 con 0x7fa8a0071de0 2024-03-29T11:23:40.417 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.437+0000 7fa89d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:23:40.417 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.437+0000 7fa89d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:40.417 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.437+0000 7fa89d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:40.417 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.437+0000 7fa89d7fa640 10 monclient(hunting): my global_id is 4341 2024-03-29T11:23:40.417 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.437+0000 7fa89d7fa640 1 -- 172.21.15.86:0/386720371 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa884004eb0 con 0x7fa8a0071de0 2024-03-29T11:23:40.420 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.438+0000 7fa89d7fa640 1 -- 172.21.15.86:0/386720371 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3493203693 0 0) 0x7fa884004eb0 con 0x7fa8a0071de0 2024-03-29T11:23:40.420 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.438+0000 7fa89d7fa640 1 -- 172.21.15.86:0/386720371 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fa8a013aec0 con 0x7fa8a0071de0 2024-03-29T11:23:40.420 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.438+0000 7fa89d7fa640 10 client.?.objecter ms_handle_connect 0x7fa8a0145ae0 2024-03-29T11:23:40.420 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.438+0000 7fa89d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:40.420 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.438+0000 7fa89d7fa640 10 client.?.objecter ms_handle_connect 0x7fa8a01378d0 2024-03-29T11:23:40.420 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.438+0000 7fa89d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:40.420 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.438+0000 7fa89d7fa640 1 -- 172.21.15.86:0/386720371 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fa894001870 con 0x7fa8a0071de0 2024-03-29T11:23:40.420 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.438+0000 7fa89d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:40.420 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.438+0000 7fa89d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:23:40.421 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.438+0000 7fa89d7fa640 10 monclient(hunting): dump: 2024-03-29T11:23:40.421 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:40.421 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:40.421 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:40.421 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:40.421 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:40.421 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:40.421 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:40.421 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:40.421 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:40.421 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:40.421 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.438+0000 7fa89d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:40.421 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.439+0000 7fa89d7fa640 1 -- 172.21.15.86:0/386720371 <== mon.0 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 115728295 0 0) 0x7fa8a013aec0 con 0x7fa8a0071de0 2024-03-29T11:23:40.421 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.439+0000 7fa89d7fa640 1 -- 172.21.15.86:0/386720371 >> v1:172.21.15.86:6790/0 conn(0x7fa8a01378d0 legacy=0x7fa8a01e1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:40.422 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.439+0000 7fa89d7fa640 1 -- 172.21.15.86:0/386720371 >> v1:172.21.15.196:6789/0 conn(0x7fa8a0145ae0 legacy=0x7fa8a0139d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:40.422 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.439+0000 7fa89d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:23:40.422 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.439+0000 7fa89d7fa640 1 monclient: found mon.a 2024-03-29T11:23:40.422 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.439+0000 7fa89d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:40.422 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.439+0000 7fa89d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:23:40.422 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.439+0000 7fa89d7fa640 1 -- 172.21.15.86:0/386720371 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8a013bb90 con 0x7fa8a0071de0 2024-03-29T11:23:40.422 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.439+0000 7fa89d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:23:40.422 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.439+0000 7fa89d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:23:40.422 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.439+0000 7fa89d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:40.422 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.439+0000 7fa89d7fa640 1 -- 172.21.15.86:0/386720371 <== mon.0 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fa8940019d0 con 0x7fa8a0071de0 2024-03-29T11:23:40.422 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.439+0000 7fa89d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:40.422 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.439+0000 7fa89d7fa640 1 -- 172.21.15.86:0/386720371 <== mon.0 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fa894004ac0 con 0x7fa8a0071de0 2024-03-29T11:23:40.422 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.439+0000 7fa89d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:40.422 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.439+0000 7fa89d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:23:40.422 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.439+0000 7fa89d7fa640 10 monclient: dump: 2024-03-29T11:23:40.423 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:40.423 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:40.423 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:40.423 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:40.423 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:40.423 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:40.423 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:40.423 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:40.423 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:40.423 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:40.423 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.439+0000 7fa89effd640 4 set_mon_vals no callback set 2024-03-29T11:23:40.423 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.440+0000 7fa8a6bcf640 5 monclient: authenticate success, global_id 4341 2024-03-29T11:23:40.423 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.440+0000 7fa8a6bcf640 10 monclient: _renew_subs 2024-03-29T11:23:40.423 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.440+0000 7fa8a6bcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:23:40.423 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.440+0000 7fa8a6bcf640 1 -- 172.21.15.86:0/386720371 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8a013be70 con 0x7fa8a0071de0 2024-03-29T11:23:40.424 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.440+0000 7fa8a6bcf640 10 client.4341.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:40.424 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.440+0000 7fa8a6bcf640 10 monclient: _renew_subs 2024-03-29T11:23:40.424 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.440+0000 7fa8a6bcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:23:40.424 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.440+0000 7fa8a6bcf640 1 -- 172.21.15.86:0/386720371 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8a0137d80 con 0x7fa8a0071de0 2024-03-29T11:23:40.424 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.440+0000 7fa8a6bcf640 1 librados: init done 2024-03-29T11:23:40.425 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.442+0000 7fa8a6bcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:40.425 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.442+0000 7fa8a6bcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:40.425 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.442+0000 7fa8a6bcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:23:40.425 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.442+0000 7fa8a6bcf640 1 -- 172.21.15.86:0/386720371 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8a0075f10 con 0x7fa8a0071de0 2024-03-29T11:23:40.425 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.443+0000 7fa89d7fa640 1 -- 172.21.15.86:0/386720371 <== mon.0 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7fa89401aa90 con 0x7fa8a0071de0 2024-03-29T11:23:40.425 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.444+0000 7fa89ffff640 10 monclient: get_auth_request con 0x7fa88406dc80 auth_method 0 2024-03-29T11:23:40.425 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.444+0000 7fa89d7fa640 1 -- 172.21.15.86:0/386720371 <== mon.0 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7fa894054880 con 0x7fa8a0071de0 2024-03-29T11:23:40.425 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.444+0000 7fa89d7fa640 10 client.4341.objecter ms_dispatch 0x7fa8a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:40.426 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.444+0000 7fa89d7fa640 3 client.4341.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:40.426 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.444+0000 7fa89d7fa640 3 client.4341.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:40.426 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.444+0000 7fa89d7fa640 20 client.4341.objecter dump_active .. 0 homeless 2024-03-29T11:23:40.426 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.444+0000 7fa89d7fa640 10 client.4341.objecter ms_handle_connect 0x7fa88406dc80 2024-03-29T11:23:40.432 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.452+0000 7fa89d7fa640 1 -- 172.21.15.86:0/386720371 <== mon.0 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fa894050b20 con 0x7fa8a0071de0 2024-03-29T11:23:40.432 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.452+0000 7fa89d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:40.432 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.452+0000 7fa89d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:40.747 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.768+0000 7fa8a6bcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:40.747 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.768+0000 7fa8a6bcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:40.747 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.768+0000 7fa8a6bcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:23:40.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.768+0000 7fa8a6bcf640 1 -- 172.21.15.86:0/386720371 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa8a013a5c0 con 0x7fa8a0071de0 2024-03-29T11:23:40.749 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:23:40.749 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:40.750 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.769+0000 7fa89d7fa640 1 -- 172.21.15.86:0/386720371 <== mon.0 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7fa894050b20 con 0x7fa8a0071de0 2024-03-29T11:23:40.750 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.770+0000 7fa89d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:40.750 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.770+0000 7fa89d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:40.753 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.774+0000 7fa8a6bcf640 10 librados: watch_flush enter 2024-03-29T11:23:40.754 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.774+0000 7fa8a6bcf640 10 librados: watch_flush exit 2024-03-29T11:23:40.754 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.774+0000 7fa8a6bcf640 20 client.4341.objecter shutdown clearing up homeless session... 2024-03-29T11:23:40.754 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.774+0000 7fa8a6bcf640 10 client.4341.objecter successfully canceled tick 2024-03-29T11:23:40.754 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.775+0000 7fa8a6bcf640 1 -- 172.21.15.86:0/386720371 >> v1:172.21.15.196:6816/588110318 conn(0x7fa88406dc80 legacy=0x7fa884070140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:40.755 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.775+0000 7fa8a6bcf640 10 monclient: shutdown 2024-03-29T11:23:40.755 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.775+0000 7fa8a6bcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:40.755 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.775+0000 7fa8a6bcf640 1 -- 172.21.15.86:0/386720371 >> v1:172.21.15.86:6789/0 conn(0x7fa8a0071de0 legacy=0x7fa8a013c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:40.755 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.776+0000 7fa8a6bcf640 1 -- 172.21.15.86:0/386720371 shutdown_connections 2024-03-29T11:23:40.755 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.776+0000 7fa8a6bcf640 1 -- 172.21.15.86:0/386720371 >> 172.21.15.86:0/386720371 conn(0x7fa8a006b660 msgr2=0x7fa8a013f2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:40.755 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.776+0000 7fa8a6bcf640 1 -- 172.21.15.86:0/386720371 shutdown_connections 2024-03-29T11:23:40.755 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.776+0000 7fa8a6bcf640 1 -- 172.21.15.86:0/386720371 wait complete. 2024-03-29T11:23:40.755 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:40.776+0000 7fa8a6bcf640 1 librados: shutdown 2024-03-29T11:23:40.772 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:47.578 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:47.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.744+0000 7f33982fb640 10 monclient: get_monmap_and_config 2024-03-29T11:23:47.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.744+0000 7f33982fb640 10 monclient: build_initial_monmap 2024-03-29T11:23:47.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.744+0000 7f33982fb640 10 monclient: monmap: 2024-03-29T11:23:47.724 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:23:47.724 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:47.725 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:23:47.745348+0000 2024-03-29T11:23:47.725 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:23:47.745348+0000 2024-03-29T11:23:47.725 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:47.725 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:47.725 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:23:47.725 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:23:47.725 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:23:47.725 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:47.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.746+0000 7f33982fb640 1 Processor -- start 2024-03-29T11:23:47.726 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.746+0000 7f33982fb640 1 -- start start 2024-03-29T11:23:47.726 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.746+0000 7f33982fb640 10 monclient: init 2024-03-29T11:23:47.728 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.747+0000 7f33982fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:47.728 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.747+0000 7f33982fb640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:47.728 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.747+0000 7f33982fb640 10 monclient(hunting): picked mon.noname-a con 0x7f33901457f0 addr v1:172.21.15.86:6789/0 2024-03-29T11:23:47.728 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.747+0000 7f33982fb640 10 monclient(hunting): picked mon.noname-b con 0x7f339013f650 addr v1:172.21.15.86:6790/0 2024-03-29T11:23:47.728 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.747+0000 7f3395896640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f339013f650 0x7f33900b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:35738/0 (socket says 172.21.15.86:35738) 2024-03-29T11:23:47.728 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.747+0000 7f33982fb640 10 monclient(hunting): picked mon.noname-c con 0x7f33900a9d70 addr v1:172.21.15.196:6789/0 2024-03-29T11:23:47.728 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.747+0000 7f3395896640 1 -- 172.21.15.86:0/3317640180 learned_addr learned my addr 172.21.15.86:0/3317640180 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:23:47.728 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.747+0000 7f33982fb640 1 -- 172.21.15.86:0/3317640180 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33900aaef0 con 0x7f33901457f0 2024-03-29T11:23:47.728 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.748+0000 7f33982fb640 1 -- 172.21.15.86:0/3317640180 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3390145bd0 con 0x7f33900a9d70 2024-03-29T11:23:47.728 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.748+0000 7f33982fb640 1 -- 172.21.15.86:0/3317640180 --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3390145eb0 con 0x7f339013f650 2024-03-29T11:23:47.728 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.748+0000 7f33982fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:47.729 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.748+0000 7f33982fb640 10 monclient(hunting): authenticate will time out at 1052.066307s 2024-03-29T11:23:47.729 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.748+0000 7f3395095640 1 -- 172.21.15.86:0/3317640180 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4203686211 0 0) 0x7f3390145eb0 con 0x7f339013f650 2024-03-29T11:23:47.729 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.748+0000 7f3395095640 10 monclient(hunting): _negotiate 2024-03-29T11:23:47.729 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.748+0000 7f3395095640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:47.729 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.748+0000 7f3395095640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:47.730 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.748+0000 7f3395095640 10 monclient(hunting): my global_id is 4364 2024-03-29T11:23:47.730 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.748+0000 7f3395095640 1 -- 172.21.15.86:0/3317640180 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3378003520 con 0x7f339013f650 2024-03-29T11:23:47.730 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.749+0000 7f3395095640 1 -- 172.21.15.86:0/3317640180 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4191689544 0 0) 0x7f33900aaef0 con 0x7f33901457f0 2024-03-29T11:23:47.730 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.749+0000 7f3395095640 10 monclient(hunting): _negotiate 2024-03-29T11:23:47.730 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.749+0000 7f3395095640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:47.730 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.749+0000 7f3395095640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:47.730 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.749+0000 7f3395095640 10 monclient(hunting): my global_id is 4344 2024-03-29T11:23:47.730 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.749+0000 7f3395095640 1 -- 172.21.15.86:0/3317640180 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3390145eb0 con 0x7f33901457f0 2024-03-29T11:23:47.730 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.749+0000 7f3395095640 1 -- 172.21.15.86:0/3317640180 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2010134051 0 0) 0x7f3378003520 con 0x7f339013f650 2024-03-29T11:23:47.730 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.749+0000 7f3395095640 1 -- 172.21.15.86:0/3317640180 >> v1:172.21.15.196:6789/0 conn(0x7f33900a9d70 legacy=0x7f339014d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:23:47.732 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.750+0000 7f3395095640 1 -- 172.21.15.86:0/3317640180 >> v1:172.21.15.86:6789/0 conn(0x7f33901457f0 legacy=0x7f339013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:47.732 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.750+0000 7f3395095640 10 monclient: _finish_hunting 0 2024-03-29T11:23:47.732 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.750+0000 7f3395095640 1 monclient: found mon.noname-b 2024-03-29T11:23:47.732 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.750+0000 7f3395095640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:47.732 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.750+0000 7f3395095640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:23:47.732 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.750+0000 7f3395095640 1 -- 172.21.15.86:0/3317640180 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3390146340 con 0x7f339013f650 2024-03-29T11:23:47.732 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.750+0000 7f3395095640 10 monclient: _finish_auth 0 2024-03-29T11:23:47.732 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.750+0000 7f3395095640 10 monclient: _check_auth_tickets 2024-03-29T11:23:47.732 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.750+0000 7f3395095640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:47.732 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.750+0000 7f3395095640 1 -- 172.21.15.86:0/3317640180 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f3384001690 con 0x7f339013f650 2024-03-29T11:23:47.732 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.750+0000 7f3395095640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:47.732 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.750+0000 7f3395095640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:23:47.732 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.750+0000 7f3395095640 10 monclient: dump: 2024-03-29T11:23:47.733 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:47.733 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:47.733 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:47.733 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:47.733 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:47.733 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:47.733 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:47.733 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:47.733 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:47.733 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:47.733 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.750+0000 7f33982fb640 5 monclient: authenticate success, global_id 4364 2024-03-29T11:23:47.733 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.750+0000 7f33982fb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:47.733 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.750+0000 7f3395095640 1 -- 172.21.15.86:0/3317640180 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3384001870 con 0x7f339013f650 2024-03-29T11:23:47.733 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.751+0000 7f3395095640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:47.734 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.751+0000 7f33982fb640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:47.734 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.751+0000 7f33982fb640 4 set_mon_vals no callback set 2024-03-29T11:23:47.734 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.751+0000 7f3395095640 1 -- 172.21.15.86:0/3317640180 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f3384001cc0 con 0x7f339013f650 2024-03-29T11:23:47.734 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.751+0000 7f3395095640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:47.734 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.751+0000 7f3395095640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:47.734 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.751+0000 7f3395095640 10 monclient: dump: 2024-03-29T11:23:47.734 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:47.734 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:47.734 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:47.734 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:47.734 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:47.734 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:47.734 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:47.734 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:47.734 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:47.735 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:47.735 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.752+0000 7f33982fb640 10 monclient: shutdown 2024-03-29T11:23:47.735 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.752+0000 7f33982fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:47.735 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.752+0000 7f33982fb640 1 -- 172.21.15.86:0/3317640180 >> v1:172.21.15.86:6790/0 conn(0x7f339013f650 legacy=0x7f33900b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:47.742 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.753+0000 7f33982fb640 1 -- 172.21.15.86:0/3317640180 shutdown_connections 2024-03-29T11:23:47.743 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.753+0000 7f33982fb640 1 -- 172.21.15.86:0/3317640180 >> 172.21.15.86:0/3317640180 conn(0x7f3390066a00 msgr2=0x7f3390066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:47.743 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.753+0000 7f33982fb640 1 -- 172.21.15.86:0/3317640180 shutdown_connections 2024-03-29T11:23:47.743 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.753+0000 7f33982fb640 1 -- 172.21.15.86:0/3317640180 wait complete. 2024-03-29T11:23:47.743 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.754+0000 7f33982fb640 10 monclient: build_initial_monmap 2024-03-29T11:23:47.743 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.754+0000 7f33982fb640 10 monclient: monmap: 2024-03-29T11:23:47.743 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:23:47.743 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:47.743 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:23:47.743 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:23:47.743 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:47.743 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:47.743 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:23:47.743 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:23:47.744 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:23:47.744 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:47.744 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.754+0000 7f33982fb640 1 librados: starting msgr at 2024-03-29T11:23:47.744 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.754+0000 7f33982fb640 1 librados: starting objecter 2024-03-29T11:23:47.744 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.754+0000 7f33982fb640 1 Processor -- start 2024-03-29T11:23:47.744 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.754+0000 7f33982fb640 1 -- start start 2024-03-29T11:23:47.744 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.754+0000 7f33982fb640 1 librados: setting wanted keys 2024-03-29T11:23:47.744 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.754+0000 7f33982fb640 1 librados: calling monclient init 2024-03-29T11:23:47.744 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.754+0000 7f33982fb640 10 monclient: init 2024-03-29T11:23:47.744 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.755+0000 7f33982fb640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:47.744 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.755+0000 7f33982fb640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:23:47.744 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.755+0000 7f33982fb640 10 monclient(hunting): picked mon.noname-b con 0x7f33900a9d70 addr v1:172.21.15.196:6789/0 2024-03-29T11:23:47.744 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.756+0000 7f3395896640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f339013f650 0x7f3390137cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:48836/0 (socket says 172.21.15.86:48836) 2024-03-29T11:23:47.744 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.756+0000 7f3395896640 1 -- 172.21.15.86:0/1197465930 learned_addr learned my addr 172.21.15.86:0/1197465930 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:23:47.744 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.756+0000 7f33982fb640 10 monclient(hunting): picked mon.noname-c con 0x7f339013f650 addr v1:172.21.15.86:6790/0 2024-03-29T11:23:47.745 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.756+0000 7f338effd640 10 client.?.objecter ms_handle_connect 0x7f339013f650 2024-03-29T11:23:47.745 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.756+0000 7f338effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:47.745 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.756+0000 7f33982fb640 10 monclient(hunting): picked mon.noname-a con 0x7f33901457f0 addr v1:172.21.15.86:6789/0 2024-03-29T11:23:47.745 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.756+0000 7f33982fb640 1 -- 172.21.15.86:0/1197465930 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33900ad390 con 0x7f33901457f0 2024-03-29T11:23:47.745 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.756+0000 7f33982fb640 1 -- 172.21.15.86:0/1197465930 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f339013bc10 con 0x7f33900a9d70 2024-03-29T11:23:47.745 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.756+0000 7f33982fb640 1 -- 172.21.15.86:0/1197465930 --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33901d7d70 con 0x7f339013f650 2024-03-29T11:23:47.745 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.756+0000 7f33982fb640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:47.745 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.756+0000 7f33982fb640 10 monclient(hunting): authenticate will time out at 1052.074717s 2024-03-29T11:23:47.745 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.756+0000 7f338effd640 1 -- 172.21.15.86:0/1197465930 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2862389468 0 0) 0x7f33901d7d70 con 0x7f339013f650 2024-03-29T11:23:47.745 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.756+0000 7f338effd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:47.745 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.756+0000 7f338effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:47.745 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.756+0000 7f338effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:47.745 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.756+0000 7f338effd640 10 monclient(hunting): my global_id is 4367 2024-03-29T11:23:47.745 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.757+0000 7f338effd640 1 -- 172.21.15.86:0/1197465930 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f337c004eb0 con 0x7f339013f650 2024-03-29T11:23:47.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.757+0000 7f338effd640 10 client.?.objecter ms_handle_connect 0x7f33901457f0 2024-03-29T11:23:47.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.757+0000 7f338effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:47.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.757+0000 7f338effd640 1 -- 172.21.15.86:0/1197465930 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3190964803 0 0) 0x7f33900ad390 con 0x7f33901457f0 2024-03-29T11:23:47.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.757+0000 7f338effd640 10 monclient(hunting): _negotiate 2024-03-29T11:23:47.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.757+0000 7f338effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:47.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.757+0000 7f338effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:47.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.757+0000 7f338effd640 10 monclient(hunting): my global_id is 4347 2024-03-29T11:23:47.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.757+0000 7f338effd640 1 -- 172.21.15.86:0/1197465930 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33901d7d70 con 0x7f33901457f0 2024-03-29T11:23:47.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.757+0000 7f338effd640 1 -- 172.21.15.86:0/1197465930 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 68421762 0 0) 0x7f337c004eb0 con 0x7f339013f650 2024-03-29T11:23:47.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.757+0000 7f338effd640 1 -- 172.21.15.86:0/1197465930 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33900ad390 con 0x7f339013f650 2024-03-29T11:23:47.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.757+0000 7f338effd640 1 -- 172.21.15.86:0/1197465930 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f3384001b80 con 0x7f339013f650 2024-03-29T11:23:47.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.757+0000 7f338effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:47.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.757+0000 7f338effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:47.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.757+0000 7f338effd640 10 monclient(hunting): dump: 2024-03-29T11:23:47.747 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:47.747 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:47.747 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:47.747 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:47.747 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:47.747 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:47.747 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:47.747 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:47.747 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:47.747 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:47.747 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.757+0000 7f338effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:47.747 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.757+0000 7f338effd640 1 -- 172.21.15.86:0/1197465930 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1532881026 0 0) 0x7f33900ad390 con 0x7f339013f650 2024-03-29T11:23:47.747 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.757+0000 7f338effd640 1 -- 172.21.15.86:0/1197465930 >> v1:172.21.15.196:6789/0 conn(0x7f33900a9d70 legacy=0x7f33901d7660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:23:47.747 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.758+0000 7f338effd640 1 -- 172.21.15.86:0/1197465930 >> v1:172.21.15.86:6789/0 conn(0x7f33901457f0 legacy=0x7f33901e2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:47.747 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.759+0000 7f338effd640 10 monclient: _finish_hunting 0 2024-03-29T11:23:47.747 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.759+0000 7f338effd640 1 monclient: found mon.c 2024-03-29T11:23:47.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.759+0000 7f338effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:47.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.759+0000 7f338effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:23:47.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.759+0000 7f338effd640 1 -- 172.21.15.86:0/1197465930 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33901d7f00 con 0x7f339013f650 2024-03-29T11:23:47.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.759+0000 7f338effd640 10 monclient: _finish_auth 0 2024-03-29T11:23:47.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.759+0000 7f338effd640 10 monclient: _check_auth_tickets 2024-03-29T11:23:47.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.759+0000 7f338effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:47.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.759+0000 7f33982fb640 5 monclient: authenticate success, global_id 4367 2024-03-29T11:23:47.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.759+0000 7f33982fb640 10 monclient: _renew_subs 2024-03-29T11:23:47.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.759+0000 7f33982fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:23:47.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.759+0000 7f33982fb640 1 -- 172.21.15.86:0/1197465930 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3390138cc0 con 0x7f339013f650 2024-03-29T11:23:47.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.759+0000 7f33982fb640 10 client.4367.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:47.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.759+0000 7f33982fb640 10 monclient: _renew_subs 2024-03-29T11:23:47.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.759+0000 7f33982fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:23:47.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.759+0000 7f33982fb640 1 -- 172.21.15.86:0/1197465930 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f33900aba70 con 0x7f339013f650 2024-03-29T11:23:47.749 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.759+0000 7f33982fb640 1 librados: init done 2024-03-29T11:23:47.749 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.759+0000 7f338effd640 1 -- 172.21.15.86:0/1197465930 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3384004bf0 con 0x7f339013f650 2024-03-29T11:23:47.749 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.759+0000 7f338effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:47.749 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.759+0000 7f338effd640 1 -- 172.21.15.86:0/1197465930 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f3384004f00 con 0x7f339013f650 2024-03-29T11:23:47.749 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.759+0000 7f338effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:47.749 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.759+0000 7f338effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:47.749 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.759+0000 7f3394894640 4 set_mon_vals no callback set 2024-03-29T11:23:47.749 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.759+0000 7f338effd640 10 monclient: dump: 2024-03-29T11:23:47.749 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:47.749 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:47.749 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:47.749 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:47.749 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:47.749 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:47.749 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:47.750 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:47.750 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:47.750 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:47.750 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.762+0000 7f338effd640 1 -- 172.21.15.86:0/1197465930 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f338401aed0 con 0x7f339013f650 2024-03-29T11:23:47.750 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.762+0000 7f338effd640 1 -- 172.21.15.86:0/1197465930 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f3384084c50 con 0x7f339013f650 2024-03-29T11:23:47.750 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.762+0000 7f338effd640 10 client.4367.objecter ms_dispatch 0x7f3390000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:47.750 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.762+0000 7f338effd640 3 client.4367.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:47.750 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.762+0000 7f338effd640 3 client.4367.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:47.750 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.762+0000 7f338effd640 20 client.4367.objecter dump_active .. 0 homeless 2024-03-29T11:23:47.750 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.762+0000 7f33982fb640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:47.750 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.762+0000 7f33982fb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:47.750 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.763+0000 7f33982fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:23:47.750 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.763+0000 7f33982fb640 1 -- 172.21.15.86:0/1197465930 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3370005080 con 0x7f339013f650 2024-03-29T11:23:47.751 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.764+0000 7f3396097640 10 monclient: get_auth_request con 0x7f337c06db40 auth_method 0 2024-03-29T11:23:47.751 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.765+0000 7f338effd640 10 client.4367.objecter ms_handle_connect 0x7f337c06db40 2024-03-29T11:23:47.751 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.770+0000 7f338effd640 1 -- 172.21.15.86:0/1197465930 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3384050530 con 0x7f339013f650 2024-03-29T11:23:47.751 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.770+0000 7f338effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:47.751 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:47.770+0000 7f338effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:48.068 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:48.087+0000 7f33982fb640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:48.068 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:48.087+0000 7f33982fb640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:48.068 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:48.087+0000 7f33982fb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:23:48.068 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:48.087+0000 7f33982fb640 1 -- 172.21.15.86:0/1197465930 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3370005330 con 0x7f339013f650 2024-03-29T11:23:48.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:48.090+0000 7f338effd640 1 -- 172.21.15.86:0/1197465930 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f3384052460 con 0x7f339013f650 2024-03-29T11:23:48.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:48.090+0000 7f338effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:48.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:48.090+0000 7f338effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:48.070 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:23:48.070 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:48.073 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:48.093+0000 7f33982fb640 10 librados: watch_flush enter 2024-03-29T11:23:48.074 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:48.093+0000 7f33982fb640 10 librados: watch_flush exit 2024-03-29T11:23:48.074 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:48.093+0000 7f33982fb640 20 client.4367.objecter shutdown clearing up homeless session... 2024-03-29T11:23:48.074 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:48.093+0000 7f33982fb640 10 client.4367.objecter successfully canceled tick 2024-03-29T11:23:48.074 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:48.093+0000 7f33982fb640 1 -- 172.21.15.86:0/1197465930 >> v1:172.21.15.196:6816/588110318 conn(0x7f337c06db40 legacy=0x7f337c070000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:48.074 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:48.093+0000 7f33982fb640 10 monclient: shutdown 2024-03-29T11:23:48.074 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:48.093+0000 7f33982fb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:48.074 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:48.093+0000 7f33982fb640 1 -- 172.21.15.86:0/1197465930 >> v1:172.21.15.86:6790/0 conn(0x7f339013f650 legacy=0x7f3390137cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:48.074 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:48.093+0000 7f33982fb640 1 -- 172.21.15.86:0/1197465930 shutdown_connections 2024-03-29T11:23:48.074 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:48.093+0000 7f33982fb640 1 -- 172.21.15.86:0/1197465930 >> 172.21.15.86:0/1197465930 conn(0x7f3390066a00 msgr2=0x7f3390066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:48.074 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:48.094+0000 7f33982fb640 1 -- 172.21.15.86:0/1197465930 shutdown_connections 2024-03-29T11:23:48.074 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:48.094+0000 7f33982fb640 1 -- 172.21.15.86:0/1197465930 wait complete. 2024-03-29T11:23:48.075 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:48.095+0000 7f33982fb640 1 librados: shutdown 2024-03-29T11:23:48.092 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:23:54.900 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:23:55.063 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.082+0000 7ff6e54f1640 10 monclient: get_monmap_and_config 2024-03-29T11:23:55.063 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.082+0000 7ff6e54f1640 10 monclient: build_initial_monmap 2024-03-29T11:23:55.063 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.082+0000 7ff6e54f1640 10 monclient: monmap: 2024-03-29T11:23:55.063 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:23:55.063 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:55.063 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:23:55.083456+0000 2024-03-29T11:23:55.063 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:23:55.083456+0000 2024-03-29T11:23:55.063 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:55.063 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:55.063 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:23:55.063 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:23:55.064 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:23:55.064 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:55.068 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.083+0000 7ff6e54f1640 1 Processor -- start 2024-03-29T11:23:55.068 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.083+0000 7ff6e54f1640 1 -- start start 2024-03-29T11:23:55.069 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.083+0000 7ff6e54f1640 10 monclient: init 2024-03-29T11:23:55.069 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.084+0000 7ff6e54f1640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:55.069 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.084+0000 7ff6e54f1640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:23:55.069 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.084+0000 7ff6e54f1640 10 monclient(hunting): picked mon.noname-b con 0x7ff6e01422b0 addr v1:172.21.15.86:6790/0 2024-03-29T11:23:55.069 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.084+0000 7ff6e54f1640 10 monclient(hunting): picked mon.noname-c con 0x7ff6e0142690 addr v1:172.21.15.196:6789/0 2024-03-29T11:23:55.069 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.085+0000 7ff6e54f1640 10 monclient(hunting): picked mon.noname-a con 0x7ff6e00b17d0 addr v1:172.21.15.86:6789/0 2024-03-29T11:23:55.069 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.085+0000 7ff6deffd640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7ff6e01422b0 0x7ff6e013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:58214/0 (socket says 172.21.15.86:58214) 2024-03-29T11:23:55.069 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.085+0000 7ff6deffd640 1 -- 172.21.15.86:0/604714874 learned_addr learned my addr 172.21.15.86:0/604714874 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:23:55.069 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.085+0000 7ff6e54f1640 1 -- 172.21.15.86:0/604714874 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e0142c80 con 0x7ff6e00b17d0 2024-03-29T11:23:55.069 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.085+0000 7ff6e54f1640 1 -- 172.21.15.86:0/604714874 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e00aae40 con 0x7ff6e0142690 2024-03-29T11:23:55.069 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.085+0000 7ff6e54f1640 1 -- 172.21.15.86:0/604714874 --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e013fbd0 con 0x7ff6e01422b0 2024-03-29T11:23:55.069 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.085+0000 7ff6e54f1640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:55.069 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.085+0000 7ff6e54f1640 10 monclient(hunting): authenticate will time out at 1059.403683s 2024-03-29T11:23:55.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.086+0000 7ff6ddffb640 1 -- 172.21.15.86:0/604714874 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2542168097 0 0) 0x7ff6e00aae40 con 0x7ff6e0142690 2024-03-29T11:23:55.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.086+0000 7ff6ddffb640 10 monclient(hunting): _negotiate 2024-03-29T11:23:55.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.086+0000 7ff6ddffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:55.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.086+0000 7ff6ddffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:55.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.086+0000 7ff6ddffb640 10 monclient(hunting): my global_id is 4318 2024-03-29T11:23:55.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.086+0000 7ff6ddffb640 1 -- 172.21.15.86:0/604714874 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6c8003520 con 0x7ff6e0142690 2024-03-29T11:23:55.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.086+0000 7ff6ddffb640 1 -- 172.21.15.86:0/604714874 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3113293447 0 0) 0x7ff6c8003520 con 0x7ff6e0142690 2024-03-29T11:23:55.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.086+0000 7ff6ddffb640 1 -- 172.21.15.86:0/604714874 >> v1:172.21.15.86:6790/0 conn(0x7ff6e01422b0 legacy=0x7ff6e013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:55.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.086+0000 7ff6ddffb640 1 -- 172.21.15.86:0/604714874 >> v1:172.21.15.86:6789/0 conn(0x7ff6e00b17d0 legacy=0x7ff6e00b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:55.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.086+0000 7ff6ddffb640 10 monclient: _finish_hunting 0 2024-03-29T11:23:55.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.086+0000 7ff6ddffb640 1 monclient: found mon.noname-c 2024-03-29T11:23:55.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.086+0000 7ff6ddffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:55.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.086+0000 7ff6ddffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T11:23:55.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.086+0000 7ff6ddffb640 1 -- 172.21.15.86:0/604714874 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e014f920 con 0x7ff6e0142690 2024-03-29T11:23:55.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.086+0000 7ff6ddffb640 10 monclient: _finish_auth 0 2024-03-29T11:23:55.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.086+0000 7ff6ddffb640 10 monclient: _check_auth_tickets 2024-03-29T11:23:55.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.086+0000 7ff6ddffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:55.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.086+0000 7ff6ddffb640 1 -- 172.21.15.86:0/604714874 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7ff6d4001690 con 0x7ff6e0142690 2024-03-29T11:23:55.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.086+0000 7ff6ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:55.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.086+0000 7ff6ddffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:55.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.086+0000 7ff6ddffb640 10 monclient: dump: 2024-03-29T11:23:55.071 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:55.072 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:55.072 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:55.072 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:55.072 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:55.072 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:55.072 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:55.072 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:55.072 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:55.072 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:55.072 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.086+0000 7ff6e54f1640 5 monclient: authenticate success, global_id 4318 2024-03-29T11:23:55.072 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.086+0000 7ff6e54f1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:23:55.073 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.087+0000 7ff6ddffb640 1 -- 172.21.15.86:0/604714874 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6d4001920 con 0x7ff6e0142690 2024-03-29T11:23:55.073 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.087+0000 7ff6ddffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:55.073 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.087+0000 7ff6ddffb640 1 -- 172.21.15.86:0/604714874 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7ff6d4001cf0 con 0x7ff6e0142690 2024-03-29T11:23:55.073 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.087+0000 7ff6ddffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:55.073 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.087+0000 7ff6ddffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:55.073 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.087+0000 7ff6ddffb640 10 monclient: dump: 2024-03-29T11:23:55.073 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:55.073 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:55.073 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:55.073 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:55.073 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:55.073 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:55.073 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:55.073 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:55.073 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:55.074 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:55.074 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.087+0000 7ff6e54f1640 10 monclient: get_monmap_and_config success 2024-03-29T11:23:55.074 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.087+0000 7ff6e54f1640 4 set_mon_vals no callback set 2024-03-29T11:23:55.074 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.087+0000 7ff6e54f1640 10 monclient: shutdown 2024-03-29T11:23:55.074 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.088+0000 7ff6e54f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:55.074 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.088+0000 7ff6e54f1640 1 -- 172.21.15.86:0/604714874 >> v1:172.21.15.196:6789/0 conn(0x7ff6e0142690 legacy=0x7ff6e00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:55.074 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.088+0000 7ff6e54f1640 1 -- 172.21.15.86:0/604714874 shutdown_connections 2024-03-29T11:23:55.074 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.088+0000 7ff6e54f1640 1 -- 172.21.15.86:0/604714874 >> 172.21.15.86:0/604714874 conn(0x7ff6e0066a00 msgr2=0x7ff6e0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:55.074 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.088+0000 7ff6e54f1640 1 -- 172.21.15.86:0/604714874 shutdown_connections 2024-03-29T11:23:55.074 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.088+0000 7ff6e54f1640 1 -- 172.21.15.86:0/604714874 wait complete. 2024-03-29T11:23:55.082 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.089+0000 7ff6e54f1640 10 monclient: build_initial_monmap 2024-03-29T11:23:55.082 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.089+0000 7ff6e54f1640 10 monclient: monmap: 2024-03-29T11:23:55.083 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:23:55.083 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:23:55.083 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:23:55.083 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:23:55.083 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:23:55.083 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:55.083 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:23:55.083 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:23:55.083 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:23:55.083 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:55.083 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.090+0000 7ff6e54f1640 1 librados: starting msgr at 2024-03-29T11:23:55.083 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.090+0000 7ff6e54f1640 1 librados: starting objecter 2024-03-29T11:23:55.083 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.090+0000 7ff6e54f1640 1 Processor -- start 2024-03-29T11:23:55.083 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.090+0000 7ff6e54f1640 1 -- start start 2024-03-29T11:23:55.083 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.090+0000 7ff6e54f1640 1 librados: setting wanted keys 2024-03-29T11:23:55.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.090+0000 7ff6e54f1640 1 librados: calling monclient init 2024-03-29T11:23:55.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.090+0000 7ff6e54f1640 10 monclient: init 2024-03-29T11:23:55.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.091+0000 7ff6e54f1640 10 monclient: _reopen_session rank -1 2024-03-29T11:23:55.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.091+0000 7ff6e54f1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:23:55.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.091+0000 7ff6e54f1640 10 monclient(hunting): picked mon.noname-a con 0x7ff6e00b17d0 addr v1:172.21.15.86:6789/0 2024-03-29T11:23:55.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.091+0000 7ff6e54f1640 10 monclient(hunting): picked mon.noname-c con 0x7ff6e01422b0 addr v1:172.21.15.86:6790/0 2024-03-29T11:23:55.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.091+0000 7ff6deffd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7ff6e00b17d0 0x7ff6e01dafc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:43514/0 (socket says 172.21.15.86:43514) 2024-03-29T11:23:55.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.092+0000 7ff6de7fc640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7ff6e01422b0 0x7ff6e01db6d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:58412/0 (socket says 172.21.15.86:58412) 2024-03-29T11:23:55.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.092+0000 7ff6df7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7ff6e0142690 0x7ff6e01e2570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:47650/0 (socket says 172.21.15.86:47650) 2024-03-29T11:23:55.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.092+0000 7ff6e54f1640 10 monclient(hunting): picked mon.noname-b con 0x7ff6e0142690 addr v1:172.21.15.196:6789/0 2024-03-29T11:23:55.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.092+0000 7ff6e54f1640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e0131600 con 0x7ff6e00b17d0 2024-03-29T11:23:55.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.092+0000 7ff6e54f1640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e00ac9f0 con 0x7ff6e0142690 2024-03-29T11:23:55.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.092+0000 7ff6e54f1640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6e0130870 con 0x7ff6e01422b0 2024-03-29T11:23:55.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.092+0000 7ff6e54f1640 10 monclient(hunting): _renew_subs 2024-03-29T11:23:55.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.092+0000 7ff6e54f1640 10 monclient(hunting): authenticate will time out at 1059.410206s 2024-03-29T11:23:55.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.092+0000 7ff6deffd640 1 -- 172.21.15.86:0/3112334316 learned_addr learned my addr 172.21.15.86:0/3112334316 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:23:55.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.092+0000 7ff6c77fe640 10 client.?.objecter ms_handle_connect 0x7ff6e00b17d0 2024-03-29T11:23:55.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.092+0000 7ff6c77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:55.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.092+0000 7ff6c77fe640 10 client.?.objecter ms_handle_connect 0x7ff6e0142690 2024-03-29T11:23:55.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.092+0000 7ff6c77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:55.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.092+0000 7ff6c77fe640 10 client.?.objecter ms_handle_connect 0x7ff6e01422b0 2024-03-29T11:23:55.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.092+0000 7ff6c77fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:23:55.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.092+0000 7ff6c77fe640 1 -- 172.21.15.86:0/3112334316 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2276710383 0 0) 0x7ff6e00ac9f0 con 0x7ff6e0142690 2024-03-29T11:23:55.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.092+0000 7ff6c77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:55.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.092+0000 7ff6c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:55.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.092+0000 7ff6c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:55.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.092+0000 7ff6c77fe640 10 monclient(hunting): my global_id is 4321 2024-03-29T11:23:55.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.092+0000 7ff6c77fe640 1 -- 172.21.15.86:0/3112334316 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6cc004eb0 con 0x7ff6e0142690 2024-03-29T11:23:55.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.092+0000 7ff6c77fe640 1 -- 172.21.15.86:0/3112334316 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2795745228 0 0) 0x7ff6e0130870 con 0x7ff6e01422b0 2024-03-29T11:23:55.086 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.092+0000 7ff6c77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:55.086 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.092+0000 7ff6c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:55.086 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.092+0000 7ff6c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:55.086 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.092+0000 7ff6c77fe640 10 monclient(hunting): my global_id is 4373 2024-03-29T11:23:55.086 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.093+0000 7ff6c77fe640 1 -- 172.21.15.86:0/3112334316 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6e00ac9f0 con 0x7ff6e01422b0 2024-03-29T11:23:55.086 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.093+0000 7ff6c77fe640 1 -- 172.21.15.86:0/3112334316 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2638910260 0 0) 0x7ff6e00ac9f0 con 0x7ff6e01422b0 2024-03-29T11:23:55.086 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.093+0000 7ff6c77fe640 1 -- 172.21.15.86:0/3112334316 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6e0130870 con 0x7ff6e01422b0 2024-03-29T11:23:55.086 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.093+0000 7ff6c77fe640 1 -- 172.21.15.86:0/3112334316 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7ff6d4001650 con 0x7ff6e01422b0 2024-03-29T11:23:55.086 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.093+0000 7ff6c77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:55.087 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.093+0000 7ff6c77fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:23:55.087 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.093+0000 7ff6c77fe640 10 monclient(hunting): dump: 2024-03-29T11:23:55.087 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:55.087 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:55.087 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:55.087 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:55.087 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:55.087 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:55.087 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:55.087 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:55.087 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:55.087 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:55.087 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.093+0000 7ff6c77fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:23:55.087 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.093+0000 7ff6c77fe640 1 -- 172.21.15.86:0/3112334316 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3993162846 0 0) 0x7ff6cc004eb0 con 0x7ff6e0142690 2024-03-29T11:23:55.088 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.093+0000 7ff6c77fe640 1 -- 172.21.15.86:0/3112334316 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7ff6e00ac9f0 con 0x7ff6e0142690 2024-03-29T11:23:55.088 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.093+0000 7ff6c77fe640 1 -- 172.21.15.86:0/3112334316 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 21705090 0 0) 0x7ff6e0131600 con 0x7ff6e00b17d0 2024-03-29T11:23:55.088 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.093+0000 7ff6c77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:23:55.088 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.093+0000 7ff6c77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:23:55.088 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.093+0000 7ff6c77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:23:55.088 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.093+0000 7ff6c77fe640 10 monclient(hunting): my global_id is 4350 2024-03-29T11:23:55.088 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.093+0000 7ff6c77fe640 1 -- 172.21.15.86:0/3112334316 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6cc004eb0 con 0x7ff6e00b17d0 2024-03-29T11:23:55.088 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.093+0000 7ff6c77fe640 1 -- 172.21.15.86:0/3112334316 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7ff6d8001be0 con 0x7ff6e0142690 2024-03-29T11:23:55.088 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.093+0000 7ff6c77fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:55.088 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.093+0000 7ff6c77fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:23:55.088 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.093+0000 7ff6c77fe640 10 monclient(hunting): dump: 2024-03-29T11:23:55.088 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:55.088 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:55.088 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:55.089 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:55.089 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:55.089 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:55.089 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:55.089 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:55.089 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:55.089 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:55.089 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.093+0000 7ff6c77fe640 1 -- 172.21.15.86:0/3112334316 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4252393133 0 0) 0x7ff6e0130870 con 0x7ff6e01422b0 2024-03-29T11:23:55.089 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.094+0000 7ff6c77fe640 1 -- 172.21.15.86:0/3112334316 >> v1:172.21.15.196:6789/0 conn(0x7ff6e0142690 legacy=0x7ff6e01e2570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:55.089 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.094+0000 7ff6c77fe640 1 -- 172.21.15.86:0/3112334316 >> v1:172.21.15.86:6789/0 conn(0x7ff6e00b17d0 legacy=0x7ff6e01dafc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:55.089 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.094+0000 7ff6c77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:23:55.089 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.094+0000 7ff6c77fe640 1 monclient: found mon.c 2024-03-29T11:23:55.089 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.094+0000 7ff6c77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:23:55.089 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.094+0000 7ff6c77fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:23:55.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.094+0000 7ff6c77fe640 1 -- 172.21.15.86:0/3112334316 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6e0130a00 con 0x7ff6e01422b0 2024-03-29T11:23:55.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.094+0000 7ff6c77fe640 10 monclient: _finish_auth 0 2024-03-29T11:23:55.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.094+0000 7ff6c77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:23:55.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.094+0000 7ff6c77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:23:55.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.094+0000 7ff6e54f1640 5 monclient: authenticate success, global_id 4373 2024-03-29T11:23:55.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.094+0000 7ff6e54f1640 10 monclient: _renew_subs 2024-03-29T11:23:55.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.094+0000 7ff6e54f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:23:55.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.094+0000 7ff6e54f1640 1 -- 172.21.15.86:0/3112334316 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6e00ab540 con 0x7ff6e01422b0 2024-03-29T11:23:55.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.094+0000 7ff6e54f1640 10 client.4373.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:23:55.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.094+0000 7ff6e54f1640 10 monclient: _renew_subs 2024-03-29T11:23:55.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.094+0000 7ff6e54f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:23:55.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.094+0000 7ff6e54f1640 1 -- 172.21.15.86:0/3112334316 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6e0134170 con 0x7ff6e01422b0 2024-03-29T11:23:55.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.094+0000 7ff6e54f1640 1 librados: init done 2024-03-29T11:23:55.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.094+0000 7ff6c77fe640 1 -- 172.21.15.86:0/3112334316 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7ff6d40038a0 con 0x7ff6e01422b0 2024-03-29T11:23:55.091 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.094+0000 7ff6c77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:23:55.091 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.094+0000 7ff6c77fe640 1 -- 172.21.15.86:0/3112334316 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7ff6d4003c70 con 0x7ff6e01422b0 2024-03-29T11:23:55.091 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.094+0000 7ff6c77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:23:55.091 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.094+0000 7ff6dd7fa640 4 set_mon_vals no callback set 2024-03-29T11:23:55.091 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.095+0000 7ff6c77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:23:55.091 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.095+0000 7ff6c77fe640 10 monclient: dump: 2024-03-29T11:23:55.091 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:23:55.091 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:23:55.091 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:55.091 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:23:55.091 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:23:55.091 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:23:55.091 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:23:55.091 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:23:55.091 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:23:55.092 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:23:55.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.096+0000 7ff6e54f1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:55.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.096+0000 7ff6e54f1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:55.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.096+0000 7ff6e54f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:23:55.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.096+0000 7ff6e54f1640 1 -- 172.21.15.86:0/3112334316 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6c0005080 con 0x7ff6e01422b0 2024-03-29T11:23:55.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.098+0000 7ff6c77fe640 1 -- 172.21.15.86:0/3112334316 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7ff6d4003dd0 con 0x7ff6e01422b0 2024-03-29T11:23:55.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.099+0000 7ff6deffd640 10 monclient: get_auth_request con 0x7ff6cc06e000 auth_method 0 2024-03-29T11:23:55.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.099+0000 7ff6c77fe640 1 -- 172.21.15.86:0/3112334316 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7ff6d40849d0 con 0x7ff6e01422b0 2024-03-29T11:23:55.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.099+0000 7ff6c77fe640 10 client.4373.objecter ms_dispatch 0x7ff6e0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:23:55.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.099+0000 7ff6c77fe640 3 client.4373.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:23:55.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.099+0000 7ff6c77fe640 3 client.4373.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:23:55.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.100+0000 7ff6c77fe640 20 client.4373.objecter dump_active .. 0 homeless 2024-03-29T11:23:55.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.100+0000 7ff6c77fe640 10 client.4373.objecter ms_handle_connect 0x7ff6cc06e000 2024-03-29T11:23:55.093 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.109+0000 7ff6c77fe640 1 -- 172.21.15.86:0/3112334316 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7ff6d40502b0 con 0x7ff6e01422b0 2024-03-29T11:23:55.093 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.109+0000 7ff6c77fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:23:55.093 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.109+0000 7ff6c77fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:23:55.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.480+0000 7ff6e54f1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:55.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.480+0000 7ff6e54f1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:55.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.480+0000 7ff6e54f1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:23:55.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.480+0000 7ff6e54f1640 1 -- 172.21.15.86:0/3112334316 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff6c0005330 con 0x7ff6e01422b0 2024-03-29T11:23:55.463 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.482+0000 7ff6c77fe640 1 -- 172.21.15.86:0/3112334316 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7ff6d40521e0 con 0x7ff6e01422b0 2024-03-29T11:23:55.463 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.482+0000 7ff6c77fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:23:55.464 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.482+0000 7ff6c77fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:23:55.464 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:23:55.464 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:23:55.468 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.486+0000 7ff6e54f1640 10 librados: watch_flush enter 2024-03-29T11:23:55.468 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.486+0000 7ff6e54f1640 10 librados: watch_flush exit 2024-03-29T11:23:55.468 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.486+0000 7ff6e54f1640 20 client.4373.objecter shutdown clearing up homeless session... 2024-03-29T11:23:55.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.486+0000 7ff6e54f1640 10 client.4373.objecter successfully canceled tick 2024-03-29T11:23:55.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.486+0000 7ff6e54f1640 1 -- 172.21.15.86:0/3112334316 >> v1:172.21.15.196:6816/588110318 conn(0x7ff6cc06e000 legacy=0x7ff6cc0704c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:55.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.486+0000 7ff6e54f1640 10 monclient: shutdown 2024-03-29T11:23:55.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.486+0000 7ff6e54f1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:23:55.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.486+0000 7ff6e54f1640 1 -- 172.21.15.86:0/3112334316 >> v1:172.21.15.86:6790/0 conn(0x7ff6e01422b0 legacy=0x7ff6e01db6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:23:55.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.486+0000 7ff6e54f1640 1 -- 172.21.15.86:0/3112334316 shutdown_connections 2024-03-29T11:23:55.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.486+0000 7ff6e54f1640 1 -- 172.21.15.86:0/3112334316 >> 172.21.15.86:0/3112334316 conn(0x7ff6e0066a00 msgr2=0x7ff6e013d670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:23:55.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.486+0000 7ff6e54f1640 1 -- 172.21.15.86:0/3112334316 shutdown_connections 2024-03-29T11:23:55.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.487+0000 7ff6e54f1640 1 -- 172.21.15.86:0/3112334316 wait complete. 2024-03-29T11:23:55.470 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:23:55.487+0000 7ff6e54f1640 1 librados: shutdown 2024-03-29T11:23:55.483 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:02.291 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:02.438 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.457+0000 7f72c2e02640 10 monclient: get_monmap_and_config 2024-03-29T11:24:02.438 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.457+0000 7f72c2e02640 10 monclient: build_initial_monmap 2024-03-29T11:24:02.438 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.457+0000 7f72c2e02640 10 monclient: monmap: 2024-03-29T11:24:02.438 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:24:02.438 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:02.438 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:24:02.458463+0000 2024-03-29T11:24:02.438 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:24:02.458463+0000 2024-03-29T11:24:02.439 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:02.439 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:02.439 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:24:02.439 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:24:02.439 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:24:02.439 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:02.439 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.458+0000 7f72c2e02640 1 Processor -- start 2024-03-29T11:24:02.440 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.459+0000 7f72c2e02640 1 -- start start 2024-03-29T11:24:02.440 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.459+0000 7f72c2e02640 10 monclient: init 2024-03-29T11:24:02.442 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.460+0000 7f72c2e02640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:02.442 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.460+0000 7f72c2e02640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:02.442 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.460+0000 7f72c2e02640 10 monclient(hunting): picked mon.noname-b con 0x7f72bc1422b0 addr v1:172.21.15.86:6790/0 2024-03-29T11:24:02.442 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.460+0000 7f72c2e02640 10 monclient(hunting): picked mon.noname-c con 0x7f72bc142690 addr v1:172.21.15.196:6789/0 2024-03-29T11:24:02.442 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.460+0000 7f72c2e02640 10 monclient(hunting): picked mon.noname-a con 0x7f72bc0b17d0 addr v1:172.21.15.86:6789/0 2024-03-29T11:24:02.442 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.460+0000 7f72c2e02640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72bc142c80 con 0x7f72bc0b17d0 2024-03-29T11:24:02.442 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.460+0000 7f72c2e02640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72bc0aae40 con 0x7f72bc142690 2024-03-29T11:24:02.442 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.460+0000 7f72c2e02640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72bc13fbd0 con 0x7f72bc1422b0 2024-03-29T11:24:02.443 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.460+0000 7f72c2e02640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:02.443 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.460+0000 7f72c2e02640 10 monclient(hunting): authenticate will time out at 1066.778699s 2024-03-29T11:24:02.443 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.460+0000 7f72bbfff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f72bc142690 0x7f72bc0b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:56802/0 (socket says 172.21.15.86:56802) 2024-03-29T11:24:02.443 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.460+0000 7f72bbfff640 1 -- 172.21.15.86:0/3398147659 learned_addr learned my addr 172.21.15.86:0/3398147659 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:24:02.443 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.461+0000 7f72bb7fe640 1 -- 172.21.15.86:0/3398147659 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3464987687 0 0) 0x7f72bc142c80 con 0x7f72bc0b17d0 2024-03-29T11:24:02.443 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.461+0000 7f72bb7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:02.443 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.461+0000 7f72bb7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:02.443 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.461+0000 7f72bb7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:02.443 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.461+0000 7f72bb7fe640 10 monclient(hunting): my global_id is 4353 2024-03-29T11:24:02.443 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.461+0000 7f72bb7fe640 1 -- 172.21.15.86:0/3398147659 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72ac003520 con 0x7f72bc0b17d0 2024-03-29T11:24:02.443 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.461+0000 7f72bb7fe640 1 -- 172.21.15.86:0/3398147659 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 607755437 0 0) 0x7f72ac003520 con 0x7f72bc0b17d0 2024-03-29T11:24:02.443 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.461+0000 7f72bb7fe640 1 -- 172.21.15.86:0/3398147659 >> v1:172.21.15.86:6790/0 conn(0x7f72bc1422b0 legacy=0x7f72bc13f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:02.446 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.464+0000 7f72bb7fe640 1 -- 172.21.15.86:0/3398147659 >> v1:172.21.15.196:6789/0 conn(0x7f72bc142690 legacy=0x7f72bc0b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:02.446 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.464+0000 7f72bb7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:02.446 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.464+0000 7f72bb7fe640 1 monclient: found mon.noname-a 2024-03-29T11:24:02.446 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.464+0000 7f72bb7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:02.446 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.464+0000 7f72bb7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.86:6789/0 2024-03-29T11:24:02.447 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.464+0000 7f72bb7fe640 1 -- 172.21.15.86:0/3398147659 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72bc151940 con 0x7f72bc0b17d0 2024-03-29T11:24:02.447 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.464+0000 7f72bb7fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:02.447 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.464+0000 7f72bb7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:02.447 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.464+0000 7f72bb7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:02.447 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.464+0000 7f72bb7fe640 1 -- 172.21.15.86:0/3398147659 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f72b4001a10 con 0x7f72bc0b17d0 2024-03-29T11:24:02.447 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.464+0000 7f72bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:02.447 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.464+0000 7f72bb7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:02.447 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.464+0000 7f72bb7fe640 10 monclient: dump: 2024-03-29T11:24:02.447 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:24:02.447 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:02.447 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:02.447 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:02.447 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:02.447 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:02.447 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:24:02.448 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:24:02.448 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:24:02.448 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:02.448 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.464+0000 7f72c2e02640 5 monclient: authenticate success, global_id 4353 2024-03-29T11:24:02.448 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.464+0000 7f72c2e02640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:02.448 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.464+0000 7f72bb7fe640 1 -- 172.21.15.86:0/3398147659 <== mon.0 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72b4001c30 con 0x7f72bc0b17d0 2024-03-29T11:24:02.448 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.464+0000 7f72bb7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:02.448 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.464+0000 7f72bb7fe640 1 -- 172.21.15.86:0/3398147659 <== mon.0 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f72b4001fc0 con 0x7f72bc0b17d0 2024-03-29T11:24:02.448 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.464+0000 7f72bb7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:02.448 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.464+0000 7f72bb7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:02.448 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.464+0000 7f72bb7fe640 10 monclient: dump: 2024-03-29T11:24:02.448 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:24:02.448 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:02.449 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:02.449 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:02.449 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:02.449 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:02.449 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:24:02.449 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:24:02.449 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:24:02.449 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:02.449 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.464+0000 7f72c2e02640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:02.449 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.464+0000 7f72c2e02640 4 set_mon_vals no callback set 2024-03-29T11:24:02.449 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.465+0000 7f72c2e02640 10 monclient: shutdown 2024-03-29T11:24:02.449 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.465+0000 7f72c2e02640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:02.449 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.465+0000 7f72c2e02640 1 -- 172.21.15.86:0/3398147659 >> v1:172.21.15.86:6789/0 conn(0x7f72bc0b17d0 legacy=0x7f72bc0b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:02.449 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.465+0000 7f72c2e02640 1 -- 172.21.15.86:0/3398147659 shutdown_connections 2024-03-29T11:24:02.449 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.465+0000 7f72c2e02640 1 -- 172.21.15.86:0/3398147659 >> 172.21.15.86:0/3398147659 conn(0x7f72bc066a00 msgr2=0x7f72bc066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:02.449 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.465+0000 7f72c2e02640 1 -- 172.21.15.86:0/3398147659 shutdown_connections 2024-03-29T11:24:02.450 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.465+0000 7f72c2e02640 1 -- 172.21.15.86:0/3398147659 wait complete. 2024-03-29T11:24:02.452 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.467+0000 7f72c2e02640 10 monclient: build_initial_monmap 2024-03-29T11:24:02.452 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.467+0000 7f72c2e02640 10 monclient: monmap: 2024-03-29T11:24:02.452 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:24:02.452 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:02.452 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:24:02.452 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:24:02.452 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:02.452 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:02.452 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:24:02.452 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:24:02.452 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:24:02.452 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:02.453 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.467+0000 7f72c2e02640 1 librados: starting msgr at 2024-03-29T11:24:02.453 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.467+0000 7f72c2e02640 1 librados: starting objecter 2024-03-29T11:24:02.453 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.467+0000 7f72c2e02640 1 Processor -- start 2024-03-29T11:24:02.453 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.467+0000 7f72c2e02640 1 -- start start 2024-03-29T11:24:02.453 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.467+0000 7f72c2e02640 1 librados: setting wanted keys 2024-03-29T11:24:02.453 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.468+0000 7f72c2e02640 1 librados: calling monclient init 2024-03-29T11:24:02.453 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.468+0000 7f72c2e02640 10 monclient: init 2024-03-29T11:24:02.453 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.469+0000 7f72c2e02640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:02.453 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.469+0000 7f72c2e02640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:02.453 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.469+0000 7f72c2e02640 10 monclient(hunting): picked mon.noname-c con 0x7f72bc0b17d0 addr v1:172.21.15.86:6790/0 2024-03-29T11:24:02.453 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.471+0000 7f72c2e02640 10 monclient(hunting): picked mon.noname-a con 0x7f72bc1422b0 addr v1:172.21.15.86:6789/0 2024-03-29T11:24:02.453 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.471+0000 7f72c2e02640 10 monclient(hunting): picked mon.noname-b con 0x7f72bc142690 addr v1:172.21.15.196:6789/0 2024-03-29T11:24:02.453 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.471+0000 7f72c2e02640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72bc0accd0 con 0x7f72bc1422b0 2024-03-29T11:24:02.453 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.471+0000 7f72c2e02640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72bc0acfb0 con 0x7f72bc142690 2024-03-29T11:24:02.453 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.471+0000 7f72c2e02640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72bc1326d0 con 0x7f72bc0b17d0 2024-03-29T11:24:02.454 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.471+0000 7f72c2e02640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:02.454 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.471+0000 7f72c2e02640 10 monclient(hunting): authenticate will time out at 1066.789086s 2024-03-29T11:24:02.454 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.471+0000 7f72bbfff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f72bc1422b0 0x7f72bc1304d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:50534/0 (socket says 172.21.15.86:50534) 2024-03-29T11:24:02.454 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.471+0000 7f72bbfff640 1 -- 172.21.15.86:0/1503561387 learned_addr learned my addr 172.21.15.86:0/1503561387 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:24:02.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.472+0000 7f72b97fa640 10 client.?.objecter ms_handle_connect 0x7f72bc1422b0 2024-03-29T11:24:02.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.472+0000 7f72b97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:02.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.472+0000 7f72b97fa640 1 -- 172.21.15.86:0/1503561387 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1157557318 0 0) 0x7f72bc0accd0 con 0x7f72bc1422b0 2024-03-29T11:24:02.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.472+0000 7f72b97fa640 10 monclient(hunting): _negotiate 2024-03-29T11:24:02.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.472+0000 7f72b97fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:02.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.472+0000 7f72b97fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:02.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.472+0000 7f72b97fa640 10 monclient(hunting): my global_id is 4356 2024-03-29T11:24:02.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.472+0000 7f72b97fa640 1 -- 172.21.15.86:0/1503561387 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72a0004eb0 con 0x7f72bc1422b0 2024-03-29T11:24:02.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.472+0000 7f72b97fa640 1 -- 172.21.15.86:0/1503561387 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1947054351 0 0) 0x7f72a0004eb0 con 0x7f72bc1422b0 2024-03-29T11:24:02.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.472+0000 7f72b97fa640 1 -- 172.21.15.86:0/1503561387 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72bc0accd0 con 0x7f72bc1422b0 2024-03-29T11:24:02.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.472+0000 7f72b97fa640 1 -- 172.21.15.86:0/1503561387 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f72a8001850 con 0x7f72bc1422b0 2024-03-29T11:24:02.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.472+0000 7f72b97fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:02.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.472+0000 7f72b97fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:02.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.472+0000 7f72b97fa640 10 monclient(hunting): dump: 2024-03-29T11:24:02.460 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:24:02.460 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:02.460 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:02.460 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:02.461 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:02.461 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:02.461 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:24:02.461 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:24:02.461 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:24:02.461 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:02.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.472+0000 7f72b97fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:02.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.472+0000 7f72b97fa640 10 client.?.objecter ms_handle_connect 0x7f72bc142690 2024-03-29T11:24:02.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.472+0000 7f72b97fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:02.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.473+0000 7f72b97fa640 1 -- 172.21.15.86:0/1503561387 <== mon.0 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3676259409 0 0) 0x7f72bc0accd0 con 0x7f72bc1422b0 2024-03-29T11:24:02.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.473+0000 7f72b97fa640 1 -- 172.21.15.86:0/1503561387 >> v1:172.21.15.86:6790/0 conn(0x7f72bc0b17d0 legacy=0x7f72bc12fdc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:02.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.473+0000 7f72b97fa640 1 -- 172.21.15.86:0/1503561387 >> v1:172.21.15.196:6789/0 conn(0x7f72bc142690 legacy=0x7f72bc1e1e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:02.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.473+0000 7f72b97fa640 10 monclient: _finish_hunting 0 2024-03-29T11:24:02.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.473+0000 7f72b97fa640 1 monclient: found mon.a 2024-03-29T11:24:02.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.473+0000 7f72b97fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:02.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.474+0000 7f72b97fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:24:02.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.474+0000 7f72b97fa640 1 -- 172.21.15.86:0/1503561387 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72bc1320d0 con 0x7f72bc1422b0 2024-03-29T11:24:02.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.474+0000 7f72b97fa640 10 monclient: _finish_auth 0 2024-03-29T11:24:02.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.474+0000 7f72b97fa640 10 monclient: _check_auth_tickets 2024-03-29T11:24:02.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.474+0000 7f72b97fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:02.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.474+0000 7f72c2e02640 5 monclient: authenticate success, global_id 4356 2024-03-29T11:24:02.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.474+0000 7f72c2e02640 10 monclient: _renew_subs 2024-03-29T11:24:02.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.474+0000 7f72c2e02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:24:02.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.474+0000 7f72c2e02640 1 -- 172.21.15.86:0/1503561387 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72bc0ab7b0 con 0x7f72bc1422b0 2024-03-29T11:24:02.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.474+0000 7f72c2e02640 10 client.4356.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:02.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.474+0000 7f72c2e02640 10 monclient: _renew_subs 2024-03-29T11:24:02.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.474+0000 7f72c2e02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:24:02.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.474+0000 7f72c2e02640 1 -- 172.21.15.86:0/1503561387 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f72bc0abb50 con 0x7f72bc1422b0 2024-03-29T11:24:02.463 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.474+0000 7f72c2e02640 1 librados: init done 2024-03-29T11:24:02.463 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.474+0000 7f72b97fa640 1 -- 172.21.15.86:0/1503561387 <== mon.0 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72a8001a20 con 0x7f72bc1422b0 2024-03-29T11:24:02.463 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.474+0000 7f72b97fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:02.463 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.474+0000 7f72b97fa640 1 -- 172.21.15.86:0/1503561387 <== mon.0 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f72a8004a00 con 0x7f72bc1422b0 2024-03-29T11:24:02.463 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.474+0000 7f72b97fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:02.463 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.474+0000 7f72b97fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:02.463 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.474+0000 7f72b97fa640 10 monclient: dump: 2024-03-29T11:24:02.463 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:24:02.463 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:02.463 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:02.463 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:02.463 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:02.463 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:02.463 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:24:02.464 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:24:02.464 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:24:02.464 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:02.464 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.474+0000 7f72ba7fc640 4 set_mon_vals no callback set 2024-03-29T11:24:02.464 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.476+0000 7f72c2e02640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:02.464 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.476+0000 7f72c2e02640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:02.464 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.476+0000 7f72c2e02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:24:02.464 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.477+0000 7f72b97fa640 1 -- 172.21.15.86:0/1503561387 <== mon.0 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f72a8004b60 con 0x7f72bc1422b0 2024-03-29T11:24:02.464 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.476+0000 7f72c2e02640 1 -- 172.21.15.86:0/1503561387 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f729c005080 con 0x7f72bc1422b0 2024-03-29T11:24:02.464 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.477+0000 7f72b97fa640 1 -- 172.21.15.86:0/1503561387 <== mon.0 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f72a8084a50 con 0x7f72bc1422b0 2024-03-29T11:24:02.464 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.477+0000 7f72b97fa640 10 client.4356.objecter ms_dispatch 0x7f72bc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:02.464 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.477+0000 7f72b97fa640 3 client.4356.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:02.464 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.478+0000 7f72b97fa640 3 client.4356.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:02.464 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.478+0000 7f72b97fa640 20 client.4356.objecter dump_active .. 0 homeless 2024-03-29T11:24:02.465 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.477+0000 7f72c0b9e640 10 monclient: get_auth_request con 0x7f72a006dbd0 auth_method 0 2024-03-29T11:24:02.465 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.478+0000 7f72b97fa640 10 client.4356.objecter ms_handle_connect 0x7f72a006dbd0 2024-03-29T11:24:02.465 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.484+0000 7f72b97fa640 1 -- 172.21.15.86:0/1503561387 <== mon.0 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f72a8050330 con 0x7f72bc1422b0 2024-03-29T11:24:02.465 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.485+0000 7f72b97fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:02.465 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.485+0000 7f72b97fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:02.799 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.818+0000 7f72c2e02640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:02.799 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.818+0000 7f72c2e02640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:02.799 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.818+0000 7f72c2e02640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:24:02.799 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.818+0000 7f72c2e02640 1 -- 172.21.15.86:0/1503561387 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f729c005330 con 0x7f72bc1422b0 2024-03-29T11:24:02.800 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.819+0000 7f72b97fa640 1 -- 172.21.15.86:0/1503561387 <== mon.0 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f72a8052260 con 0x7f72bc1422b0 2024-03-29T11:24:02.800 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.819+0000 7f72b97fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:02.800 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.819+0000 7f72b97fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:02.800 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:24:02.801 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:02.804 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.823+0000 7f72c2e02640 10 librados: watch_flush enter 2024-03-29T11:24:02.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.823+0000 7f72c2e02640 10 librados: watch_flush exit 2024-03-29T11:24:02.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.823+0000 7f72c2e02640 20 client.4356.objecter shutdown clearing up homeless session... 2024-03-29T11:24:02.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.823+0000 7f72c2e02640 10 client.4356.objecter successfully canceled tick 2024-03-29T11:24:02.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.823+0000 7f72c2e02640 1 -- 172.21.15.86:0/1503561387 >> v1:172.21.15.196:6816/588110318 conn(0x7f72a006dbd0 legacy=0x7f72a0070090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:02.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.823+0000 7f72c2e02640 10 monclient: shutdown 2024-03-29T11:24:02.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.823+0000 7f72c2e02640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:02.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.823+0000 7f72c2e02640 1 -- 172.21.15.86:0/1503561387 >> v1:172.21.15.86:6789/0 conn(0x7f72bc1422b0 legacy=0x7f72bc1304d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:02.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.824+0000 7f72c2e02640 1 -- 172.21.15.86:0/1503561387 shutdown_connections 2024-03-29T11:24:02.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.824+0000 7f72c2e02640 1 -- 172.21.15.86:0/1503561387 >> 172.21.15.86:0/1503561387 conn(0x7f72bc066a00 msgr2=0x7f72bc0b3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:02.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.824+0000 7f72c2e02640 1 -- 172.21.15.86:0/1503561387 shutdown_connections 2024-03-29T11:24:02.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.824+0000 7f72c2e02640 1 -- 172.21.15.86:0/1503561387 wait complete. 2024-03-29T11:24:02.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:02.824+0000 7f72c2e02640 1 librados: shutdown 2024-03-29T11:24:02.820 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:09.627 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:09.780 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.798+0000 7fc357dcb640 10 monclient: get_monmap_and_config 2024-03-29T11:24:09.780 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.798+0000 7fc357dcb640 10 monclient: build_initial_monmap 2024-03-29T11:24:09.780 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.798+0000 7fc357dcb640 10 monclient: monmap: 2024-03-29T11:24:09.780 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:24:09.780 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:09.781 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:24:09.799822+0000 2024-03-29T11:24:09.781 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:24:09.799822+0000 2024-03-29T11:24:09.781 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:09.781 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:09.781 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:24:09.781 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:24:09.781 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:24:09.781 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:09.782 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.800+0000 7fc357dcb640 1 Processor -- start 2024-03-29T11:24:09.782 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.800+0000 7fc357dcb640 1 -- start start 2024-03-29T11:24:09.782 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.800+0000 7fc357dcb640 10 monclient: init 2024-03-29T11:24:09.782 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.801+0000 7fc357dcb640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:09.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.801+0000 7fc357dcb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:09.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.801+0000 7fc357dcb640 10 monclient(hunting): picked mon.noname-c con 0x7fc3501457f0 addr v1:172.21.15.196:6789/0 2024-03-29T11:24:09.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.802+0000 7fc357dcb640 10 monclient(hunting): picked mon.noname-a con 0x7fc35013f650 addr v1:172.21.15.86:6789/0 2024-03-29T11:24:09.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.802+0000 7fc357dcb640 10 monclient(hunting): picked mon.noname-b con 0x7fc3500a9d70 addr v1:172.21.15.86:6790/0 2024-03-29T11:24:09.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.802+0000 7fc357dcb640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3500aaef0 con 0x7fc35013f650 2024-03-29T11:24:09.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.802+0000 7fc357dcb640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc350145bd0 con 0x7fc3501457f0 2024-03-29T11:24:09.784 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.802+0000 7fc357dcb640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc350145eb0 con 0x7fc3500a9d70 2024-03-29T11:24:09.784 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.802+0000 7fc357dcb640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:09.784 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.802+0000 7fc357dcb640 10 monclient(hunting): authenticate will time out at 1074.120863s 2024-03-29T11:24:09.785 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.804+0000 7fc355366640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc35013f650 0x7fc3500b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:37440/0 (socket says 172.21.15.86:37440) 2024-03-29T11:24:09.785 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.805+0000 7fc355b67640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fc3501457f0 0x7fc35013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:43167/0 (socket says 172.21.15.86:43167) 2024-03-29T11:24:09.786 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.805+0000 7fc355b67640 1 -- 172.21.15.86:0/1015892843 learned_addr learned my addr 172.21.15.86:0/1015892843 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:24:09.788 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.807+0000 7fc354b65640 1 -- 172.21.15.86:0/1015892843 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1420528715 0 0) 0x7fc350145bd0 con 0x7fc3501457f0 2024-03-29T11:24:09.788 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.807+0000 7fc354b65640 10 monclient(hunting): _negotiate 2024-03-29T11:24:09.788 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.807+0000 7fc354b65640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:09.788 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.807+0000 7fc354b65640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:09.788 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.807+0000 7fc354b65640 10 monclient(hunting): my global_id is 4327 2024-03-29T11:24:09.788 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.807+0000 7fc354b65640 1 -- 172.21.15.86:0/1015892843 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc33c003520 con 0x7fc3501457f0 2024-03-29T11:24:09.789 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.808+0000 7fc354b65640 1 -- 172.21.15.86:0/1015892843 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 426869886 0 0) 0x7fc33c003520 con 0x7fc3501457f0 2024-03-29T11:24:09.789 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.808+0000 7fc354b65640 1 -- 172.21.15.86:0/1015892843 >> v1:172.21.15.86:6790/0 conn(0x7fc3500a9d70 legacy=0x7fc35014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:09.789 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.808+0000 7fc354b65640 1 -- 172.21.15.86:0/1015892843 >> v1:172.21.15.86:6789/0 conn(0x7fc35013f650 legacy=0x7fc3500b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:09.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.808+0000 7fc354b65640 10 monclient: _finish_hunting 0 2024-03-29T11:24:09.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.808+0000 7fc354b65640 1 monclient: found mon.noname-c 2024-03-29T11:24:09.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.808+0000 7fc354b65640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:09.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.808+0000 7fc354b65640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T11:24:09.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.808+0000 7fc354b65640 1 -- 172.21.15.86:0/1015892843 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc350146340 con 0x7fc3501457f0 2024-03-29T11:24:09.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.808+0000 7fc354b65640 10 monclient: _finish_auth 0 2024-03-29T11:24:09.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.808+0000 7fc354b65640 10 monclient: _check_auth_tickets 2024-03-29T11:24:09.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.808+0000 7fc354b65640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:09.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.808+0000 7fc354b65640 1 -- 172.21.15.86:0/1015892843 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc3400015c0 con 0x7fc3501457f0 2024-03-29T11:24:09.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.808+0000 7fc354b65640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:09.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.808+0000 7fc354b65640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:09.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.808+0000 7fc354b65640 10 monclient: dump: 2024-03-29T11:24:09.790 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:24:09.790 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:09.791 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:09.791 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:09.791 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:09.791 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:09.791 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:24:09.791 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:24:09.791 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:24:09.791 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:09.791 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.808+0000 7fc357dcb640 5 monclient: authenticate success, global_id 4327 2024-03-29T11:24:09.791 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.808+0000 7fc357dcb640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:09.791 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.808+0000 7fc354b65640 1 -- 172.21.15.86:0/1015892843 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc340001850 con 0x7fc3501457f0 2024-03-29T11:24:09.791 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.809+0000 7fc354b65640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:09.791 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.809+0000 7fc354b65640 1 -- 172.21.15.86:0/1015892843 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc340001be0 con 0x7fc3501457f0 2024-03-29T11:24:09.791 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.809+0000 7fc357dcb640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:09.791 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.809+0000 7fc357dcb640 4 set_mon_vals no callback set 2024-03-29T11:24:09.792 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.809+0000 7fc354b65640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:09.792 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.809+0000 7fc354b65640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:09.792 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.809+0000 7fc354b65640 10 monclient: dump: 2024-03-29T11:24:09.792 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:24:09.792 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:09.792 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:09.792 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:09.792 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:09.792 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:09.792 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:24:09.792 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:24:09.792 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:24:09.792 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:09.793 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.809+0000 7fc357dcb640 10 monclient: shutdown 2024-03-29T11:24:09.793 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.810+0000 7fc357dcb640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:09.794 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.810+0000 7fc357dcb640 1 -- 172.21.15.86:0/1015892843 >> v1:172.21.15.196:6789/0 conn(0x7fc3501457f0 legacy=0x7fc35013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:09.794 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.810+0000 7fc357dcb640 1 -- 172.21.15.86:0/1015892843 shutdown_connections 2024-03-29T11:24:09.794 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.810+0000 7fc357dcb640 1 -- 172.21.15.86:0/1015892843 >> 172.21.15.86:0/1015892843 conn(0x7fc350066a00 msgr2=0x7fc350066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:09.794 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.810+0000 7fc357dcb640 1 -- 172.21.15.86:0/1015892843 shutdown_connections 2024-03-29T11:24:09.794 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.811+0000 7fc357dcb640 1 -- 172.21.15.86:0/1015892843 wait complete. 2024-03-29T11:24:09.794 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.811+0000 7fc357dcb640 10 monclient: build_initial_monmap 2024-03-29T11:24:09.794 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.811+0000 7fc357dcb640 10 monclient: monmap: 2024-03-29T11:24:09.794 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:24:09.794 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:09.794 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:24:09.795 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:24:09.795 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:09.795 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:09.795 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:24:09.795 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:24:09.795 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:24:09.795 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:09.795 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.812+0000 7fc357dcb640 1 librados: starting msgr at 2024-03-29T11:24:09.795 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.812+0000 7fc357dcb640 1 librados: starting objecter 2024-03-29T11:24:09.795 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.812+0000 7fc357dcb640 1 Processor -- start 2024-03-29T11:24:09.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.812+0000 7fc357dcb640 1 -- start start 2024-03-29T11:24:09.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.812+0000 7fc357dcb640 1 librados: setting wanted keys 2024-03-29T11:24:09.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.812+0000 7fc357dcb640 1 librados: calling monclient init 2024-03-29T11:24:09.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.812+0000 7fc357dcb640 10 monclient: init 2024-03-29T11:24:09.797 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.813+0000 7fc357dcb640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:09.797 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.813+0000 7fc357dcb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:09.797 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.813+0000 7fc357dcb640 10 monclient(hunting): picked mon.noname-b con 0x7fc3500a9d70 addr v1:172.21.15.196:6789/0 2024-03-29T11:24:09.797 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.813+0000 7fc357dcb640 10 monclient(hunting): picked mon.noname-a con 0x7fc35013f650 addr v1:172.21.15.86:6789/0 2024-03-29T11:24:09.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.814+0000 7fc355366640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc35013f650 0x7fc3501dc110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:41270/0 (socket says 172.21.15.86:41270) 2024-03-29T11:24:09.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.816+0000 7fc356368640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7fc3501457f0 0x7fc3501376f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:38325/0 (socket says 172.21.15.86:38325) 2024-03-29T11:24:09.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.816+0000 7fc355366640 1 -- 172.21.15.86:0/3091158112 learned_addr learned my addr 172.21.15.86:0/3091158112 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:24:09.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.816+0000 7fc357dcb640 10 monclient(hunting): picked mon.noname-c con 0x7fc3501457f0 addr v1:172.21.15.86:6790/0 2024-03-29T11:24:09.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.816+0000 7fc357dcb640 1 -- 172.21.15.86:0/3091158112 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc35013b8a0 con 0x7fc35013f650 2024-03-29T11:24:09.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.817+0000 7fc357dcb640 1 -- 172.21.15.86:0/3091158112 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc35006dfc0 con 0x7fc3500a9d70 2024-03-29T11:24:09.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.817+0000 7fc357dcb640 1 -- 172.21.15.86:0/3091158112 --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc35006e2a0 con 0x7fc3501457f0 2024-03-29T11:24:09.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.817+0000 7fc357dcb640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:09.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.817+0000 7fc357dcb640 10 monclient(hunting): authenticate will time out at 1074.135232s 2024-03-29T11:24:09.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.817+0000 7fc3467fc640 10 client.?.objecter ms_handle_connect 0x7fc35013f650 2024-03-29T11:24:09.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.817+0000 7fc3467fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:09.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.817+0000 7fc3467fc640 10 client.?.objecter ms_handle_connect 0x7fc3501457f0 2024-03-29T11:24:09.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.817+0000 7fc3467fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:09.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.817+0000 7fc3467fc640 10 client.?.objecter ms_handle_connect 0x7fc3500a9d70 2024-03-29T11:24:09.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.817+0000 7fc3467fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:09.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.817+0000 7fc3467fc640 1 -- 172.21.15.86:0/3091158112 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2345937734 0 0) 0x7fc35013b8a0 con 0x7fc35013f650 2024-03-29T11:24:09.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.817+0000 7fc3467fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:09.807 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.817+0000 7fc3467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:09.807 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.817+0000 7fc3467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:09.807 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.817+0000 7fc3467fc640 10 monclient(hunting): my global_id is 4362 2024-03-29T11:24:09.807 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.817+0000 7fc3467fc640 1 -- 172.21.15.86:0/3091158112 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc334004eb0 con 0x7fc35013f650 2024-03-29T11:24:09.807 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.817+0000 7fc3467fc640 1 -- 172.21.15.86:0/3091158112 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 288653536 0 0) 0x7fc35006e2a0 con 0x7fc3501457f0 2024-03-29T11:24:09.807 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.817+0000 7fc3467fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:09.807 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.817+0000 7fc3467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:09.807 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.817+0000 7fc3467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:09.807 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.817+0000 7fc3467fc640 10 monclient(hunting): my global_id is 4376 2024-03-29T11:24:09.807 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.817+0000 7fc3467fc640 1 -- 172.21.15.86:0/3091158112 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc35013b8a0 con 0x7fc3501457f0 2024-03-29T11:24:09.807 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.817+0000 7fc3467fc640 1 -- 172.21.15.86:0/3091158112 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2877090523 0 0) 0x7fc35006dfc0 con 0x7fc3500a9d70 2024-03-29T11:24:09.807 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.817+0000 7fc3467fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:09.807 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.817+0000 7fc3467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:09.807 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.817+0000 7fc3467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:09.808 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.817+0000 7fc3467fc640 10 monclient(hunting): my global_id is 4330 2024-03-29T11:24:09.808 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.817+0000 7fc3467fc640 1 -- 172.21.15.86:0/3091158112 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc35006e2a0 con 0x7fc3500a9d70 2024-03-29T11:24:09.808 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.818+0000 7fc3467fc640 1 -- 172.21.15.86:0/3091158112 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3952086552 0 0) 0x7fc334004eb0 con 0x7fc35013f650 2024-03-29T11:24:09.808 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.818+0000 7fc3467fc640 1 -- 172.21.15.86:0/3091158112 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc35006dfc0 con 0x7fc35013f650 2024-03-29T11:24:09.808 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.818+0000 7fc3467fc640 1 -- 172.21.15.86:0/3091158112 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc348001d50 con 0x7fc35013f650 2024-03-29T11:24:09.808 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.818+0000 7fc3467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:09.808 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.818+0000 7fc3467fc640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:09.808 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.818+0000 7fc3467fc640 10 monclient(hunting): dump: 2024-03-29T11:24:09.808 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:24:09.808 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:09.808 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:09.808 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:09.808 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:09.808 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:09.808 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:24:09.809 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:24:09.809 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:24:09.809 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:09.809 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.818+0000 7fc3467fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:09.809 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.818+0000 7fc3467fc640 1 -- 172.21.15.86:0/3091158112 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 116876095 0 0) 0x7fc35013b8a0 con 0x7fc3501457f0 2024-03-29T11:24:09.809 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.818+0000 7fc3467fc640 1 -- 172.21.15.86:0/3091158112 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc334004eb0 con 0x7fc3501457f0 2024-03-29T11:24:09.809 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.818+0000 7fc3467fc640 1 -- 172.21.15.86:0/3091158112 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc34c0016d0 con 0x7fc3501457f0 2024-03-29T11:24:09.809 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.818+0000 7fc3467fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:09.809 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.818+0000 7fc3467fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:09.809 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.818+0000 7fc3467fc640 10 monclient(hunting): dump: 2024-03-29T11:24:09.809 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:24:09.809 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:09.809 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:09.809 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:09.809 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:09.810 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:09.810 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:24:09.810 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:24:09.810 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:24:09.810 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:09.810 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.818+0000 7fc3467fc640 1 -- 172.21.15.86:0/3091158112 <== mon.0 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3066291411 0 0) 0x7fc35006dfc0 con 0x7fc35013f650 2024-03-29T11:24:09.810 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.818+0000 7fc3467fc640 1 -- 172.21.15.86:0/3091158112 >> v1:172.21.15.86:6790/0 conn(0x7fc3501457f0 legacy=0x7fc3501376f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:09.810 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.818+0000 7fc3467fc640 1 -- 172.21.15.86:0/3091158112 >> v1:172.21.15.196:6789/0 conn(0x7fc3500a9d70 legacy=0x7fc35013a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:09.810 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.818+0000 7fc3467fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:09.810 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.818+0000 7fc3467fc640 1 monclient: found mon.a 2024-03-29T11:24:09.810 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.818+0000 7fc3467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:09.810 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.818+0000 7fc3467fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:24:09.811 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.818+0000 7fc3467fc640 1 -- 172.21.15.86:0/3091158112 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3501dbbf0 con 0x7fc35013f650 2024-03-29T11:24:09.811 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.818+0000 7fc3467fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:09.811 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.818+0000 7fc3467fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:09.811 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.818+0000 7fc3467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:09.811 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.818+0000 7fc357dcb640 5 monclient: authenticate success, global_id 4362 2024-03-29T11:24:09.811 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.818+0000 7fc357dcb640 10 monclient: _renew_subs 2024-03-29T11:24:09.811 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.818+0000 7fc357dcb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:24:09.811 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.818+0000 7fc357dcb640 1 -- 172.21.15.86:0/3091158112 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3501db880 con 0x7fc35013f650 2024-03-29T11:24:09.811 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.818+0000 7fc357dcb640 10 client.4362.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:09.811 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.818+0000 7fc357dcb640 10 monclient: _renew_subs 2024-03-29T11:24:09.811 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.818+0000 7fc357dcb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:24:09.811 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.818+0000 7fc357dcb640 1 -- 172.21.15.86:0/3091158112 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc350137fe0 con 0x7fc35013f650 2024-03-29T11:24:09.811 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.818+0000 7fc357dcb640 1 librados: init done 2024-03-29T11:24:09.811 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.819+0000 7fc3467fc640 1 -- 172.21.15.86:0/3091158112 <== mon.0 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc348004b60 con 0x7fc35013f650 2024-03-29T11:24:09.812 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.819+0000 7fc3467fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:09.812 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.819+0000 7fc347fff640 4 set_mon_vals no callback set 2024-03-29T11:24:09.812 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.820+0000 7fc333fff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:09.812 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.820+0000 7fc3467fc640 1 -- 172.21.15.86:0/3091158112 <== mon.0 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc348004f30 con 0x7fc35013f650 2024-03-29T11:24:09.812 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.820+0000 7fc3467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:09.812 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.820+0000 7fc3467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:09.812 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.820+0000 7fc3467fc640 10 monclient: dump: 2024-03-29T11:24:09.812 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:24:09.812 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:09.812 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:09.812 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:09.812 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:09.812 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:09.812 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:24:09.812 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:24:09.813 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:24:09.813 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:09.813 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.820+0000 7fc333fff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:09.813 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.820+0000 7fc333fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:24:09.813 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.820+0000 7fc333fff640 1 -- 172.21.15.86:0/3091158112 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3501dbde0 con 0x7fc35013f650 2024-03-29T11:24:09.813 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.822+0000 7fc3467fc640 1 -- 172.21.15.86:0/3091158112 <== mon.0 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7fc348005150 con 0x7fc35013f650 2024-03-29T11:24:09.813 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.822+0000 7fc3467fc640 1 -- 172.21.15.86:0/3091158112 <== mon.0 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7fc348084e90 con 0x7fc35013f650 2024-03-29T11:24:09.813 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.822+0000 7fc355b67640 10 monclient: get_auth_request con 0x7fc33406e070 auth_method 0 2024-03-29T11:24:09.813 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.822+0000 7fc3467fc640 10 client.4362.objecter ms_dispatch 0x7fc350000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:09.813 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.823+0000 7fc3467fc640 3 client.4362.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:09.813 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.823+0000 7fc3467fc640 3 client.4362.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:09.813 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.823+0000 7fc3467fc640 20 client.4362.objecter dump_active .. 0 homeless 2024-03-29T11:24:09.813 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.823+0000 7fc3467fc640 10 client.4362.objecter ms_handle_connect 0x7fc33406e070 2024-03-29T11:24:09.817 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.832+0000 7fc3467fc640 1 -- 172.21.15.86:0/3091158112 <== mon.0 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc348050770 con 0x7fc35013f650 2024-03-29T11:24:09.817 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.832+0000 7fc3467fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:09.817 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:09.832+0000 7fc3467fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:10.141 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:10.160+0000 7fc333fff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:10.142 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:10.160+0000 7fc333fff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:10.142 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:10.160+0000 7fc333fff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:24:10.142 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:10.160+0000 7fc333fff640 1 -- 172.21.15.86:0/3091158112 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3501dba10 con 0x7fc35013f650 2024-03-29T11:24:10.146 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:10.165+0000 7fc3467fc640 1 -- 172.21.15.86:0/3091158112 <== mon.0 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7fc3480526a0 con 0x7fc35013f650 2024-03-29T11:24:10.146 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:10.165+0000 7fc3467fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:10.146 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:10.165+0000 7fc3467fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:10.146 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:24:10.147 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:10.150 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:10.168+0000 7fc333fff640 10 librados: watch_flush enter 2024-03-29T11:24:10.150 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:10.168+0000 7fc333fff640 10 librados: watch_flush exit 2024-03-29T11:24:10.150 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:10.168+0000 7fc333fff640 20 client.4362.objecter shutdown clearing up homeless session... 2024-03-29T11:24:10.150 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:10.168+0000 7fc333fff640 10 client.4362.objecter successfully canceled tick 2024-03-29T11:24:10.150 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:10.168+0000 7fc333fff640 1 -- 172.21.15.86:0/3091158112 >> v1:172.21.15.196:6816/588110318 conn(0x7fc33406e070 legacy=0x7fc334070530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:10.150 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:10.169+0000 7fc333fff640 10 monclient: shutdown 2024-03-29T11:24:10.150 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:10.169+0000 7fc333fff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:10.150 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:10.169+0000 7fc333fff640 1 -- 172.21.15.86:0/3091158112 >> v1:172.21.15.86:6789/0 conn(0x7fc35013f650 legacy=0x7fc3501dc110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:10.151 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:10.169+0000 7fc333fff640 1 -- 172.21.15.86:0/3091158112 shutdown_connections 2024-03-29T11:24:10.151 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:10.169+0000 7fc333fff640 1 -- 172.21.15.86:0/3091158112 >> 172.21.15.86:0/3091158112 conn(0x7fc350066a00 msgr2=0x7fc3500b5790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:10.151 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:10.169+0000 7fc333fff640 1 -- 172.21.15.86:0/3091158112 shutdown_connections 2024-03-29T11:24:10.151 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:10.169+0000 7fc333fff640 1 -- 172.21.15.86:0/3091158112 wait complete. 2024-03-29T11:24:10.151 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:10.169+0000 7fc333fff640 1 librados: shutdown 2024-03-29T11:24:10.167 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:16.973 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:17.119 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.137+0000 7f981802a640 10 monclient: get_monmap_and_config 2024-03-29T11:24:17.119 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.137+0000 7f981802a640 10 monclient: build_initial_monmap 2024-03-29T11:24:17.119 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.137+0000 7f981802a640 10 monclient: monmap: 2024-03-29T11:24:17.119 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:24:17.119 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:17.119 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:24:17.138419+0000 2024-03-29T11:24:17.119 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:24:17.138419+0000 2024-03-29T11:24:17.119 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:17.119 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:17.119 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:24:17.119 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:24:17.119 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:24:17.120 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:17.122 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.140+0000 7f981802a640 1 Processor -- start 2024-03-29T11:24:17.123 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.140+0000 7f981802a640 1 -- start start 2024-03-29T11:24:17.123 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.140+0000 7f981802a640 10 monclient: init 2024-03-29T11:24:17.123 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.141+0000 7f981802a640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:17.123 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.141+0000 7f981802a640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:17.123 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.141+0000 7f981802a640 10 monclient(hunting): picked mon.noname-c con 0x7f98100b0860 addr v1:172.21.15.196:6789/0 2024-03-29T11:24:17.123 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.141+0000 7f981802a640 10 monclient(hunting): picked mon.noname-a con 0x7f981006d9d0 addr v1:172.21.15.86:6789/0 2024-03-29T11:24:17.123 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.141+0000 7f981802a640 10 monclient(hunting): picked mon.noname-b con 0x7f981013ef30 addr v1:172.21.15.86:6790/0 2024-03-29T11:24:17.123 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.141+0000 7f981802a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98100b0c40 con 0x7f981006d9d0 2024-03-29T11:24:17.123 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.141+0000 7f981802a640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98100aed10 con 0x7f98100b0860 2024-03-29T11:24:17.123 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.141+0000 7f981802a640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98100ab1b0 con 0x7f981013ef30 2024-03-29T11:24:17.123 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.141+0000 7f981802a640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:17.123 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.141+0000 7f981802a640 10 monclient(hunting): authenticate will time out at 1081.459368s 2024-03-29T11:24:17.124 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.141+0000 7f98165c7640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f981013ef30 0x7f98100af1d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:47092/0 (socket says 172.21.15.86:47092) 2024-03-29T11:24:17.124 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.141+0000 7f98165c7640 1 -- 172.21.15.86:0/125900625 learned_addr learned my addr 172.21.15.86:0/125900625 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:24:17.126 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.144+0000 7f9814dc4640 1 -- 172.21.15.86:0/125900625 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2944371062 0 0) 0x7f98100ab1b0 con 0x7f981013ef30 2024-03-29T11:24:17.126 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.144+0000 7f9814dc4640 10 monclient(hunting): _negotiate 2024-03-29T11:24:17.126 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.144+0000 7f9814dc4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:17.127 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.144+0000 7f9814dc4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:17.127 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.144+0000 7f9814dc4640 10 monclient(hunting): my global_id is 4379 2024-03-29T11:24:17.127 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.144+0000 7f9814dc4640 1 -- 172.21.15.86:0/125900625 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97fc003520 con 0x7f981013ef30 2024-03-29T11:24:17.127 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.145+0000 7f9814dc4640 1 -- 172.21.15.86:0/125900625 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2227750992 0 0) 0x7f98100b0c40 con 0x7f981006d9d0 2024-03-29T11:24:17.127 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.145+0000 7f9814dc4640 10 monclient(hunting): _negotiate 2024-03-29T11:24:17.127 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.145+0000 7f9814dc4640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:17.127 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.145+0000 7f9814dc4640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:17.127 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.145+0000 7f9814dc4640 10 monclient(hunting): my global_id is 4365 2024-03-29T11:24:17.130 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.145+0000 7f9814dc4640 1 -- 172.21.15.86:0/125900625 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98100ab1b0 con 0x7f981006d9d0 2024-03-29T11:24:17.130 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.145+0000 7f9814dc4640 1 -- 172.21.15.86:0/125900625 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 228261883 0 0) 0x7f97fc003520 con 0x7f981013ef30 2024-03-29T11:24:17.131 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.145+0000 7f9814dc4640 1 -- 172.21.15.86:0/125900625 >> v1:172.21.15.196:6789/0 conn(0x7f98100b0860 legacy=0x7f981013e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:17.131 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.145+0000 7f9814dc4640 1 -- 172.21.15.86:0/125900625 >> v1:172.21.15.86:6789/0 conn(0x7f981006d9d0 legacy=0x7f9810146a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:17.131 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.145+0000 7f9814dc4640 10 monclient: _finish_hunting 0 2024-03-29T11:24:17.131 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.145+0000 7f9814dc4640 1 monclient: found mon.noname-b 2024-03-29T11:24:17.131 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.145+0000 7f9814dc4640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:17.131 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.145+0000 7f9814dc4640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:24:17.131 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.145+0000 7f9814dc4640 1 -- 172.21.15.86:0/125900625 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98100ab490 con 0x7f981013ef30 2024-03-29T11:24:17.131 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.145+0000 7f9814dc4640 10 monclient: _finish_auth 0 2024-03-29T11:24:17.131 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.145+0000 7f9814dc4640 10 monclient: _check_auth_tickets 2024-03-29T11:24:17.131 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.145+0000 7f9814dc4640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:17.131 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.145+0000 7f9814dc4640 1 -- 172.21.15.86:0/125900625 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f980c001690 con 0x7f981013ef30 2024-03-29T11:24:17.131 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.145+0000 7f9814dc4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:17.131 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.145+0000 7f9814dc4640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:17.131 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.145+0000 7f9814dc4640 10 monclient: dump: 2024-03-29T11:24:17.131 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:24:17.132 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:17.132 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:17.132 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:17.132 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:17.132 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:17.132 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:24:17.132 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:24:17.132 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:24:17.132 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:17.132 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.145+0000 7f981802a640 5 monclient: authenticate success, global_id 4379 2024-03-29T11:24:17.132 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.145+0000 7f981802a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:17.132 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.145+0000 7f9814dc4640 1 -- 172.21.15.86:0/125900625 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f980c0017f0 con 0x7f981013ef30 2024-03-29T11:24:17.132 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.145+0000 7f9814dc4640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:17.132 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.145+0000 7f9814dc4640 1 -- 172.21.15.86:0/125900625 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f980c001c00 con 0x7f981013ef30 2024-03-29T11:24:17.132 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.145+0000 7f981802a640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:17.133 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.145+0000 7f981802a640 4 set_mon_vals no callback set 2024-03-29T11:24:17.133 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.145+0000 7f9814dc4640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:17.133 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.145+0000 7f9814dc4640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:17.133 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.145+0000 7f9814dc4640 10 monclient: dump: 2024-03-29T11:24:17.133 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:24:17.133 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:17.133 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:17.133 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:17.133 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:17.133 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:17.133 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:24:17.133 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:24:17.133 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:24:17.133 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:17.134 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.146+0000 7f981802a640 10 monclient: shutdown 2024-03-29T11:24:17.134 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.146+0000 7f981802a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:17.134 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.146+0000 7f981802a640 1 -- 172.21.15.86:0/125900625 >> v1:172.21.15.86:6790/0 conn(0x7f981013ef30 legacy=0x7f98100af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:17.134 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.146+0000 7f981802a640 1 -- 172.21.15.86:0/125900625 shutdown_connections 2024-03-29T11:24:17.134 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.146+0000 7f981802a640 1 -- 172.21.15.86:0/125900625 >> 172.21.15.86:0/125900625 conn(0x7f981006b660 msgr2=0x7f981006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:17.134 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.146+0000 7f981802a640 1 -- 172.21.15.86:0/125900625 shutdown_connections 2024-03-29T11:24:17.134 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.146+0000 7f981802a640 1 -- 172.21.15.86:0/125900625 wait complete. 2024-03-29T11:24:17.134 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.148+0000 7f981802a640 10 monclient: build_initial_monmap 2024-03-29T11:24:17.134 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.148+0000 7f981802a640 10 monclient: monmap: 2024-03-29T11:24:17.134 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:24:17.134 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:17.134 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:24:17.134 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:24:17.134 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:17.135 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:17.135 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:24:17.135 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:24:17.135 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:24:17.135 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:17.135 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.149+0000 7f981802a640 1 librados: starting msgr at 2024-03-29T11:24:17.135 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.149+0000 7f981802a640 1 librados: starting objecter 2024-03-29T11:24:17.135 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.149+0000 7f981802a640 1 Processor -- start 2024-03-29T11:24:17.135 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.149+0000 7f981802a640 1 -- start start 2024-03-29T11:24:17.135 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.149+0000 7f981802a640 1 librados: setting wanted keys 2024-03-29T11:24:17.135 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.149+0000 7f981802a640 1 librados: calling monclient init 2024-03-29T11:24:17.135 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.149+0000 7f981802a640 10 monclient: init 2024-03-29T11:24:17.142 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.150+0000 7f981802a640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:17.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.150+0000 7f981802a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:17.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.150+0000 7f981802a640 10 monclient(hunting): picked mon.noname-a con 0x7f981006d9d0 addr v1:172.21.15.86:6789/0 2024-03-29T11:24:17.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.150+0000 7f981802a640 10 monclient(hunting): picked mon.noname-c con 0x7f98100b0860 addr v1:172.21.15.86:6790/0 2024-03-29T11:24:17.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.150+0000 7f981802a640 10 monclient(hunting): picked mon.noname-b con 0x7f981013ef30 addr v1:172.21.15.196:6789/0 2024-03-29T11:24:17.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.150+0000 7f981802a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98100b35d0 con 0x7f981006d9d0 2024-03-29T11:24:17.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.150+0000 7f981802a640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98100b3d40 con 0x7f981013ef30 2024-03-29T11:24:17.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.150+0000 7f981802a640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98100b4260 con 0x7f98100b0860 2024-03-29T11:24:17.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.150+0000 7f981802a640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:17.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.150+0000 7f981802a640 10 monclient(hunting): authenticate will time out at 1081.468638s 2024-03-29T11:24:17.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.153+0000 7f98165c7640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f981013ef30 0x7f98101e7240 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:44205/0 (socket says 172.21.15.86:44205) 2024-03-29T11:24:17.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.153+0000 7f98165c7640 1 -- 172.21.15.86:0/1158899105 learned_addr learned my addr 172.21.15.86:0/1158899105 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:24:17.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.153+0000 7f98067fc640 10 client.?.objecter ms_handle_connect 0x7f981006d9d0 2024-03-29T11:24:17.144 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.153+0000 7f98067fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:17.144 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.153+0000 7f98067fc640 10 client.?.objecter ms_handle_connect 0x7f98100b0860 2024-03-29T11:24:17.144 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.153+0000 7f98067fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:17.144 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.154+0000 7f98067fc640 1 -- 172.21.15.86:0/1158899105 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1821509448 0 0) 0x7f98100b4260 con 0x7f98100b0860 2024-03-29T11:24:17.144 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.154+0000 7f98067fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:17.144 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.154+0000 7f98067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:17.144 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.154+0000 7f98067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:17.144 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.154+0000 7f98067fc640 10 monclient(hunting): my global_id is 4382 2024-03-29T11:24:17.144 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.154+0000 7f98067fc640 1 -- 172.21.15.86:0/1158899105 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97f4004eb0 con 0x7f98100b0860 2024-03-29T11:24:17.144 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.154+0000 7f98067fc640 10 client.?.objecter ms_handle_connect 0x7f981013ef30 2024-03-29T11:24:17.144 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.154+0000 7f98067fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:17.144 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.154+0000 7f98067fc640 1 -- 172.21.15.86:0/1158899105 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2037353637 0 0) 0x7f98100b35d0 con 0x7f981006d9d0 2024-03-29T11:24:17.144 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.154+0000 7f98067fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:17.144 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.154+0000 7f98067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:17.144 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.154+0000 7f98067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:17.145 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.154+0000 7f98067fc640 10 monclient(hunting): my global_id is 4368 2024-03-29T11:24:17.145 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.154+0000 7f98067fc640 1 -- 172.21.15.86:0/1158899105 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98100b4260 con 0x7f981006d9d0 2024-03-29T11:24:17.145 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.155+0000 7f98067fc640 1 -- 172.21.15.86:0/1158899105 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1319163751 0 0) 0x7f98100b3d40 con 0x7f981013ef30 2024-03-29T11:24:17.145 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.155+0000 7f98067fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:17.145 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.155+0000 7f98067fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:17.145 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.155+0000 7f98067fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:17.145 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.155+0000 7f98067fc640 10 monclient(hunting): my global_id is 4333 2024-03-29T11:24:17.145 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.155+0000 7f98067fc640 1 -- 172.21.15.86:0/1158899105 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f98100b35d0 con 0x7f981013ef30 2024-03-29T11:24:17.145 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.155+0000 7f98067fc640 1 -- 172.21.15.86:0/1158899105 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2983851329 0 0) 0x7f97f4004eb0 con 0x7f98100b0860 2024-03-29T11:24:17.145 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.155+0000 7f98067fc640 1 -- 172.21.15.86:0/1158899105 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f98100b3d40 con 0x7f98100b0860 2024-03-29T11:24:17.145 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.155+0000 7f98067fc640 1 -- 172.21.15.86:0/1158899105 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 182967960 0 0) 0x7f98100b4260 con 0x7f981006d9d0 2024-03-29T11:24:17.145 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.155+0000 7f98067fc640 1 -- 172.21.15.86:0/1158899105 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97f4004eb0 con 0x7f981006d9d0 2024-03-29T11:24:17.145 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.155+0000 7f98067fc640 1 -- 172.21.15.86:0/1158899105 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f98000018b0 con 0x7f98100b0860 2024-03-29T11:24:17.145 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.155+0000 7f98067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:17.146 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.155+0000 7f98067fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:17.146 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.155+0000 7f98067fc640 10 monclient(hunting): dump: 2024-03-29T11:24:17.146 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:24:17.146 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:17.146 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:17.146 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:17.146 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:17.146 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:17.146 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:24:17.146 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:24:17.146 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:24:17.146 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:17.146 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.156+0000 7f98067fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:17.146 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.156+0000 7f98067fc640 1 -- 172.21.15.86:0/1158899105 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f9808001810 con 0x7f981006d9d0 2024-03-29T11:24:17.147 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.156+0000 7f98067fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:17.147 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.156+0000 7f98067fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:17.147 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.156+0000 7f98067fc640 10 monclient(hunting): dump: 2024-03-29T11:24:17.147 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:24:17.147 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:17.147 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:17.147 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:17.147 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:17.147 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:17.147 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:24:17.147 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:24:17.147 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:24:17.147 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:17.147 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.156+0000 7f98067fc640 1 -- 172.21.15.86:0/1158899105 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1516603762 0 0) 0x7f98100b3d40 con 0x7f98100b0860 2024-03-29T11:24:17.148 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.156+0000 7f98067fc640 1 -- 172.21.15.86:0/1158899105 >> v1:172.21.15.196:6789/0 conn(0x7f981013ef30 legacy=0x7f98101e7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:17.148 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.156+0000 7f98067fc640 1 -- 172.21.15.86:0/1158899105 >> v1:172.21.15.86:6789/0 conn(0x7f981006d9d0 legacy=0x7f98100b11d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:17.148 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.156+0000 7f98067fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:17.148 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.156+0000 7f98067fc640 1 monclient: found mon.c 2024-03-29T11:24:17.148 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.156+0000 7f98067fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:17.148 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.156+0000 7f98067fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:24:17.148 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.156+0000 7f98067fc640 1 -- 172.21.15.86:0/1158899105 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98100b3960 con 0x7f98100b0860 2024-03-29T11:24:17.148 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.156+0000 7f98067fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:17.148 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.156+0000 7f98067fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:17.148 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.156+0000 7f98067fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:17.148 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.156+0000 7f981802a640 5 monclient: authenticate success, global_id 4382 2024-03-29T11:24:17.148 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.156+0000 7f981802a640 10 monclient: _renew_subs 2024-03-29T11:24:17.148 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.156+0000 7f981802a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:24:17.148 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.156+0000 7f981802a640 1 -- 172.21.15.86:0/1158899105 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f981013bc10 con 0x7f98100b0860 2024-03-29T11:24:17.148 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.156+0000 7f981802a640 10 client.4382.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:17.149 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.156+0000 7f981802a640 10 monclient: _renew_subs 2024-03-29T11:24:17.149 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.156+0000 7f981802a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:24:17.149 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.156+0000 7f981802a640 1 -- 172.21.15.86:0/1158899105 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f98100b2490 con 0x7f98100b0860 2024-03-29T11:24:17.149 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.156+0000 7f981802a640 1 librados: init done 2024-03-29T11:24:17.149 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.157+0000 7f981802a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:17.149 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.157+0000 7f981802a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:17.149 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.157+0000 7f981802a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:24:17.149 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.157+0000 7f981802a640 1 -- 172.21.15.86:0/1158899105 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98100b3bb0 con 0x7f98100b0860 2024-03-29T11:24:17.149 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.158+0000 7f98067fc640 1 -- 172.21.15.86:0/1158899105 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9800001a10 con 0x7f98100b0860 2024-03-29T11:24:17.149 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.158+0000 7f98067fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:17.149 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.159+0000 7f98067fc640 1 -- 172.21.15.86:0/1158899105 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f9800003790 con 0x7f98100b0860 2024-03-29T11:24:17.149 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.159+0000 7f9807fff640 4 set_mon_vals no callback set 2024-03-29T11:24:17.149 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.159+0000 7f98067fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:17.149 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.159+0000 7f98067fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:17.150 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.159+0000 7f98067fc640 10 monclient: dump: 2024-03-29T11:24:17.150 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:24:17.150 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:17.150 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:17.150 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:17.150 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:17.150 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:17.150 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:24:17.150 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:24:17.150 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:24:17.150 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:17.150 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.159+0000 7f98067fc640 1 -- 172.21.15.86:0/1158899105 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f98000197a0 con 0x7f98100b0860 2024-03-29T11:24:17.150 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.160+0000 7f98067fc640 1 -- 172.21.15.86:0/1158899105 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f9800083890 con 0x7f98100b0860 2024-03-29T11:24:17.150 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.160+0000 7f9815dc6640 10 monclient: get_auth_request con 0x7f97f406e1f0 auth_method 0 2024-03-29T11:24:17.151 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.160+0000 7f98067fc640 10 client.4382.objecter ms_dispatch 0x7f9810000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:17.151 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.160+0000 7f98067fc640 3 client.4382.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:17.151 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.160+0000 7f98067fc640 3 client.4382.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:17.151 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.160+0000 7f98067fc640 20 client.4382.objecter dump_active .. 0 homeless 2024-03-29T11:24:17.151 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.161+0000 7f98067fc640 10 client.4382.objecter ms_handle_connect 0x7f97f406e1f0 2024-03-29T11:24:17.151 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.168+0000 7f98067fc640 1 -- 172.21.15.86:0/1158899105 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f980004f1f0 con 0x7f98100b0860 2024-03-29T11:24:17.151 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.168+0000 7f98067fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:17.152 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.168+0000 7f98067fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:17.472 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.491+0000 7f981802a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:17.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.491+0000 7f981802a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:17.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.491+0000 7f981802a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:24:17.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.491+0000 7f981802a640 1 -- 172.21.15.86:0/1158899105 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f981013c730 con 0x7f98100b0860 2024-03-29T11:24:17.475 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.494+0000 7f98067fc640 1 -- 172.21.15.86:0/1158899105 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f9800051120 con 0x7f98100b0860 2024-03-29T11:24:17.475 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.494+0000 7f98067fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:17.475 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.494+0000 7f98067fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:17.476 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:24:17.476 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:17.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.497+0000 7f981802a640 10 librados: watch_flush enter 2024-03-29T11:24:17.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.498+0000 7f981802a640 10 librados: watch_flush exit 2024-03-29T11:24:17.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.498+0000 7f981802a640 20 client.4382.objecter shutdown clearing up homeless session... 2024-03-29T11:24:17.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.498+0000 7f981802a640 10 client.4382.objecter successfully canceled tick 2024-03-29T11:24:17.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.499+0000 7f981802a640 1 -- 172.21.15.86:0/1158899105 >> v1:172.21.15.196:6816/588110318 conn(0x7f97f406e1f0 legacy=0x7f97f40706b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:17.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.499+0000 7f981802a640 10 monclient: shutdown 2024-03-29T11:24:17.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.499+0000 7f981802a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:17.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.499+0000 7f981802a640 1 -- 172.21.15.86:0/1158899105 >> v1:172.21.15.86:6790/0 conn(0x7f98100b0860 legacy=0x7f981014eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:17.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.499+0000 7f981802a640 1 -- 172.21.15.86:0/1158899105 shutdown_connections 2024-03-29T11:24:17.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.499+0000 7f981802a640 1 -- 172.21.15.86:0/1158899105 >> 172.21.15.86:0/1158899105 conn(0x7f981006b660 msgr2=0x7f981013e5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:17.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.499+0000 7f981802a640 1 -- 172.21.15.86:0/1158899105 shutdown_connections 2024-03-29T11:24:17.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.500+0000 7f981802a640 1 -- 172.21.15.86:0/1158899105 wait complete. 2024-03-29T11:24:17.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:17.500+0000 7f981802a640 1 librados: shutdown 2024-03-29T11:24:17.498 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:24.306 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:24.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.475+0000 7f2e6efab640 10 monclient: get_monmap_and_config 2024-03-29T11:24:24.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.475+0000 7f2e6efab640 10 monclient: build_initial_monmap 2024-03-29T11:24:24.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.475+0000 7f2e6efab640 10 monclient: monmap: 2024-03-29T11:24:24.457 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:24:24.457 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:24.457 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:24:24.476200+0000 2024-03-29T11:24:24.457 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:24:24.476200+0000 2024-03-29T11:24:24.457 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:24.457 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:24.457 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:24:24.457 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:24:24.458 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:24:24.458 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:24.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.476+0000 7f2e6efab640 1 Processor -- start 2024-03-29T11:24:24.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.476+0000 7f2e6efab640 1 -- start start 2024-03-29T11:24:24.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.476+0000 7f2e6efab640 10 monclient: init 2024-03-29T11:24:24.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.477+0000 7f2e6efab640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:24.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.477+0000 7f2e6efab640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:24.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.477+0000 7f2e6efab640 10 monclient(hunting): picked mon.noname-c con 0x7f2e600389e0 addr v1:172.21.15.196:6789/0 2024-03-29T11:24:24.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.478+0000 7f2e6efab640 10 monclient(hunting): picked mon.noname-b con 0x7f2e60038dc0 addr v1:172.21.15.86:6790/0 2024-03-29T11:24:24.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.478+0000 7f2e6efab640 10 monclient(hunting): picked mon.noname-a con 0x7f2e60030660 addr v1:172.21.15.86:6789/0 2024-03-29T11:24:24.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.478+0000 7f2e6efab640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e600c54c0 con 0x7f2e60030660 2024-03-29T11:24:24.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.478+0000 7f2e6efab640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e60036430 con 0x7f2e600389e0 2024-03-29T11:24:24.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.478+0000 7f2e6efab640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e60039840 con 0x7f2e60038dc0 2024-03-29T11:24:24.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.478+0000 7f2e6efab640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:24.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.478+0000 7f2e6efab640 10 monclient(hunting): authenticate will time out at 1088.796551s 2024-03-29T11:24:24.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.478+0000 7f2e6d7a8640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f2e60038dc0 0x7f2e600cadd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:35106/0 (socket says 172.21.15.86:35106) 2024-03-29T11:24:24.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.478+0000 7f2e6d7a8640 1 -- 172.21.15.86:0/1878263800 learned_addr learned my addr 172.21.15.86:0/1878263800 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:24:24.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.479+0000 7f2e6cfa7640 1 -- 172.21.15.86:0/1878263800 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 683137789 0 0) 0x7f2e60039840 con 0x7f2e60038dc0 2024-03-29T11:24:24.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.479+0000 7f2e6cfa7640 10 monclient(hunting): _negotiate 2024-03-29T11:24:24.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.479+0000 7f2e6cfa7640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:24.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.479+0000 7f2e6cfa7640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:24.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.479+0000 7f2e6cfa7640 10 monclient(hunting): my global_id is 4385 2024-03-29T11:24:24.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.479+0000 7f2e6cfa7640 1 -- 172.21.15.86:0/1878263800 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e54003520 con 0x7f2e60038dc0 2024-03-29T11:24:24.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.480+0000 7f2e6cfa7640 1 -- 172.21.15.86:0/1878263800 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3245935114 0 0) 0x7f2e54003520 con 0x7f2e60038dc0 2024-03-29T11:24:24.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.480+0000 7f2e6cfa7640 1 -- 172.21.15.86:0/1878263800 >> v1:172.21.15.196:6789/0 conn(0x7f2e600389e0 legacy=0x7f2e60033520 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:24:24.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.480+0000 7f2e6cfa7640 1 -- 172.21.15.86:0/1878263800 >> v1:172.21.15.86:6789/0 conn(0x7f2e60030660 legacy=0x7f2e600d3170 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:24:24.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.482+0000 7f2e6cfa7640 10 monclient: _finish_hunting 0 2024-03-29T11:24:24.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.482+0000 7f2e6cfa7640 1 monclient: found mon.noname-b 2024-03-29T11:24:24.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.483+0000 7f2e6cfa7640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:24.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.483+0000 7f2e6cfa7640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:24:24.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.483+0000 7f2e6cfa7640 1 -- 172.21.15.86:0/1878263800 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e60035690 con 0x7f2e60038dc0 2024-03-29T11:24:24.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.483+0000 7f2e6cfa7640 10 monclient: _finish_auth 0 2024-03-29T11:24:24.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.483+0000 7f2e6cfa7640 10 monclient: _check_auth_tickets 2024-03-29T11:24:24.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.483+0000 7f2e6cfa7640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:24.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.483+0000 7f2e6cfa7640 1 -- 172.21.15.86:0/1878263800 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f2e5c001690 con 0x7f2e60038dc0 2024-03-29T11:24:24.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.483+0000 7f2e6cfa7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:24.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.483+0000 7f2e6cfa7640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:24.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.483+0000 7f2e6cfa7640 10 monclient: dump: 2024-03-29T11:24:24.467 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:24:24.467 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:24.467 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:24.467 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:24.467 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:24.467 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:24.467 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:24:24.467 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:24:24.467 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:24:24.467 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:24.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.483+0000 7f2e6efab640 5 monclient: authenticate success, global_id 4385 2024-03-29T11:24:24.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.483+0000 7f2e6efab640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:24.468 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.483+0000 7f2e6cfa7640 1 -- 172.21.15.86:0/1878263800 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e5c001920 con 0x7f2e60038dc0 2024-03-29T11:24:24.468 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.483+0000 7f2e6cfa7640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:24.468 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.483+0000 7f2e6cfa7640 1 -- 172.21.15.86:0/1878263800 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f2e5c001cb0 con 0x7f2e60038dc0 2024-03-29T11:24:24.468 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.483+0000 7f2e6efab640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:24.468 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.483+0000 7f2e6efab640 4 set_mon_vals no callback set 2024-03-29T11:24:24.468 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.484+0000 7f2e6cfa7640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:24.468 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.484+0000 7f2e6cfa7640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:24.468 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.484+0000 7f2e6cfa7640 10 monclient: dump: 2024-03-29T11:24:24.468 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:24:24.468 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:24.468 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:24.468 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:24.468 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:24.468 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:24.469 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:24:24.469 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:24:24.469 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:24:24.469 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:24.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.484+0000 7f2e6efab640 10 monclient: shutdown 2024-03-29T11:24:24.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.484+0000 7f2e6efab640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:24.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.484+0000 7f2e6efab640 1 -- 172.21.15.86:0/1878263800 >> v1:172.21.15.86:6790/0 conn(0x7f2e60038dc0 legacy=0x7f2e600cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:24.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.488+0000 7f2e6efab640 1 -- 172.21.15.86:0/1878263800 shutdown_connections 2024-03-29T11:24:24.470 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.488+0000 7f2e6efab640 1 -- 172.21.15.86:0/1878263800 >> 172.21.15.86:0/1878263800 conn(0x7f2e60005c00 msgr2=0x7f2e60006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:24.470 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.488+0000 7f2e6efab640 1 -- 172.21.15.86:0/1878263800 shutdown_connections 2024-03-29T11:24:24.471 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.489+0000 7f2e6efab640 1 -- 172.21.15.86:0/1878263800 wait complete. 2024-03-29T11:24:24.472 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.490+0000 7f2e6efab640 10 monclient: build_initial_monmap 2024-03-29T11:24:24.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.490+0000 7f2e6efab640 10 monclient: monmap: 2024-03-29T11:24:24.473 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:24:24.473 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:24.473 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:24:24.473 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:24:24.473 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:24.473 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:24.473 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:24:24.473 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:24:24.473 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:24:24.473 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:24.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.490+0000 7f2e6efab640 1 librados: starting msgr at 2024-03-29T11:24:24.474 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.490+0000 7f2e6efab640 1 librados: starting objecter 2024-03-29T11:24:24.474 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.491+0000 7f2e6efab640 1 Processor -- start 2024-03-29T11:24:24.474 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.491+0000 7f2e6efab640 1 -- start start 2024-03-29T11:24:24.474 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.491+0000 7f2e6efab640 1 librados: setting wanted keys 2024-03-29T11:24:24.474 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.491+0000 7f2e6efab640 1 librados: calling monclient init 2024-03-29T11:24:24.475 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.491+0000 7f2e6efab640 10 monclient: init 2024-03-29T11:24:24.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.492+0000 7f2e6efab640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:24.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.492+0000 7f2e6efab640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:24:24.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.493+0000 7f2e6dfa9640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f2e60030660 0x7f2e6002f670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:38310/0 (socket says 172.21.15.86:38310) 2024-03-29T11:24:24.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.493+0000 7f2e6dfa9640 1 -- 172.21.15.86:0/2281439370 learned_addr learned my addr 172.21.15.86:0/2281439370 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:24:24.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.493+0000 7f2e6efab640 10 monclient(hunting): picked mon.noname-c con 0x7f2e60030660 addr v1:172.21.15.86:6790/0 2024-03-29T11:24:24.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.493+0000 7f2e6efab640 10 monclient(hunting): picked mon.noname-b con 0x7f2e600389e0 addr v1:172.21.15.196:6789/0 2024-03-29T11:24:24.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.493+0000 7f2e6efab640 10 monclient(hunting): picked mon.noname-a con 0x7f2e6002ec60 addr v1:172.21.15.86:6789/0 2024-03-29T11:24:24.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.493+0000 7f2e6efab640 1 -- 172.21.15.86:0/2281439370 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e6002fed0 con 0x7f2e6002ec60 2024-03-29T11:24:24.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.495+0000 7f2e6efab640 1 -- 172.21.15.86:0/2281439370 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e6000d680 con 0x7f2e600389e0 2024-03-29T11:24:24.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.495+0000 7f2e6efab640 1 -- 172.21.15.86:0/2281439370 --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2e6000d810 con 0x7f2e60030660 2024-03-29T11:24:24.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.495+0000 7f2e6efab640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:24.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.495+0000 7f2e6efab640 10 monclient(hunting): authenticate will time out at 1088.813566s 2024-03-29T11:24:24.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.495+0000 7f2e5a7fc640 10 client.?.objecter ms_handle_connect 0x7f2e60030660 2024-03-29T11:24:24.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.495+0000 7f2e5a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:24.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.496+0000 7f2e5a7fc640 1 -- 172.21.15.86:0/2281439370 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3304185496 0 0) 0x7f2e6000d810 con 0x7f2e60030660 2024-03-29T11:24:24.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.496+0000 7f2e5a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:24.479 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.496+0000 7f2e5a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:24.479 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.496+0000 7f2e5a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:24.479 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.496+0000 7f2e5a7fc640 10 monclient(hunting): my global_id is 4388 2024-03-29T11:24:24.479 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.496+0000 7f2e5a7fc640 1 -- 172.21.15.86:0/2281439370 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2e4c004eb0 con 0x7f2e60030660 2024-03-29T11:24:24.479 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.496+0000 7f2e5a7fc640 1 -- 172.21.15.86:0/2281439370 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1368316375 0 0) 0x7f2e4c004eb0 con 0x7f2e60030660 2024-03-29T11:24:24.479 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.496+0000 7f2e5a7fc640 1 -- 172.21.15.86:0/2281439370 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2e6000d810 con 0x7f2e60030660 2024-03-29T11:24:24.479 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.496+0000 7f2e5a7fc640 1 -- 172.21.15.86:0/2281439370 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f2e64001870 con 0x7f2e60030660 2024-03-29T11:24:24.479 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.496+0000 7f2e5a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:24.479 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.496+0000 7f2e5a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:24.479 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.496+0000 7f2e5a7fc640 10 monclient(hunting): dump: 2024-03-29T11:24:24.479 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:24:24.479 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:24.479 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:24.479 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:24.479 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:24.480 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:24.480 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:24:24.480 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:24:24.480 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:24:24.480 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:24.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.496+0000 7f2e5a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:24.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.496+0000 7f2e5a7fc640 1 -- 172.21.15.86:0/2281439370 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 120967111 0 0) 0x7f2e6000d810 con 0x7f2e60030660 2024-03-29T11:24:24.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.496+0000 7f2e5a7fc640 1 -- 172.21.15.86:0/2281439370 >> v1:172.21.15.196:6789/0 conn(0x7f2e600389e0 legacy=0x7f2e6000cf70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:24.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.503+0000 7f2e5a7fc640 1 -- 172.21.15.86:0/2281439370 >> v1:172.21.15.86:6789/0 conn(0x7f2e6002ec60 legacy=0x7f2e601609d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:24.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.503+0000 7f2e5a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:24.487 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.504+0000 7f2e5a7fc640 1 monclient: found mon.c 2024-03-29T11:24:24.487 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.504+0000 7f2e5a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:24.487 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.504+0000 7f2e5a7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:24:24.487 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.504+0000 7f2e5a7fc640 1 -- 172.21.15.86:0/2281439370 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e6000d9a0 con 0x7f2e60030660 2024-03-29T11:24:24.487 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.504+0000 7f2e5a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:24.487 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.504+0000 7f2e5a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:24.487 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.504+0000 7f2e5a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:24.487 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.504+0000 7f2e5a7fc640 10 client.?.objecter ms_handle_connect 0x7f2e6002ec60 2024-03-29T11:24:24.487 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.504+0000 7f2e5a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:24.487 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.504+0000 7f2e6efab640 5 monclient: authenticate success, global_id 4388 2024-03-29T11:24:24.487 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.504+0000 7f2e6efab640 10 monclient: _renew_subs 2024-03-29T11:24:24.487 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.504+0000 7f2e6efab640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:24:24.487 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.504+0000 7f2e6efab640 1 -- 172.21.15.86:0/2281439370 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e600c7ce0 con 0x7f2e60030660 2024-03-29T11:24:24.488 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.504+0000 7f2e6efab640 10 client.4388.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:24.488 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.504+0000 7f2e6efab640 10 monclient: _renew_subs 2024-03-29T11:24:24.488 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.504+0000 7f2e5a7fc640 1 -- 172.21.15.86:0/2281439370 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2e640019d0 con 0x7f2e60030660 2024-03-29T11:24:24.488 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.504+0000 7f2e6efab640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:24:24.488 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.504+0000 7f2e6efab640 1 -- 172.21.15.86:0/2281439370 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e601612c0 con 0x7f2e60030660 2024-03-29T11:24:24.488 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.504+0000 7f2e6efab640 1 librados: init done 2024-03-29T11:24:24.488 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.504+0000 7f2e5a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:24.488 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.504+0000 7f2e5a7fc640 1 -- 172.21.15.86:0/2281439370 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f2e64004ae0 con 0x7f2e60030660 2024-03-29T11:24:24.488 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.504+0000 7f2e5a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:24.488 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.504+0000 7f2e5a7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:24.488 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.504+0000 7f2e5a7fc640 10 monclient: dump: 2024-03-29T11:24:24.488 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:24:24.488 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:24.488 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:24.488 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:24.489 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:24.489 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:24.489 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:24:24.489 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:24:24.489 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:24:24.489 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:24.489 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.504+0000 7f2e5bfff640 4 set_mon_vals no callback set 2024-03-29T11:24:24.490 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.507+0000 7f2e5a7fc640 1 -- 172.21.15.86:0/2281439370 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f2e6401aab0 con 0x7f2e60030660 2024-03-29T11:24:24.490 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.507+0000 7f2e4bfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:24.490 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.507+0000 7f2e4bfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:24.490 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.507+0000 7f2e4bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:24:24.490 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.507+0000 7f2e4bfff640 1 -- 172.21.15.86:0/2281439370 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e60007b90 con 0x7f2e60030660 2024-03-29T11:24:24.491 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.508+0000 7f2e5a7fc640 1 -- 172.21.15.86:0/2281439370 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f2e64084b10 con 0x7f2e60030660 2024-03-29T11:24:24.491 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.508+0000 7f2e5a7fc640 10 client.4388.objecter ms_dispatch 0x7f2e68000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:24.491 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.508+0000 7f2e5a7fc640 3 client.4388.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:24.491 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.508+0000 7f2e5a7fc640 3 client.4388.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:24.491 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.508+0000 7f2e5a7fc640 20 client.4388.objecter dump_active .. 0 homeless 2024-03-29T11:24:24.491 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.510+0000 7f2e6d7a8640 10 monclient: get_auth_request con 0x7f2e4c06da40 auth_method 0 2024-03-29T11:24:24.495 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.514+0000 7f2e5a7fc640 10 client.4388.objecter ms_handle_connect 0x7f2e4c06da40 2024-03-29T11:24:24.498 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.516+0000 7f2e5a7fc640 1 -- 172.21.15.86:0/2281439370 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2e640503f0 con 0x7f2e60030660 2024-03-29T11:24:24.498 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.516+0000 7f2e5a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:24.498 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.516+0000 7f2e5a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:24.838 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.857+0000 7f2e4bfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:24.839 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.857+0000 7f2e4bfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:24.839 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.857+0000 7f2e4bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:24:24.839 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.857+0000 7f2e4bfff640 1 -- 172.21.15.86:0/2281439370 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2e60005120 con 0x7f2e60030660 2024-03-29T11:24:24.839 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.858+0000 7f2e5a7fc640 1 -- 172.21.15.86:0/2281439370 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f2e64052320 con 0x7f2e60030660 2024-03-29T11:24:24.839 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.858+0000 7f2e5a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:24.839 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.858+0000 7f2e5a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:24.840 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:24:24.840 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:24.843 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.861+0000 7f2e4bfff640 10 librados: watch_flush enter 2024-03-29T11:24:24.843 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.861+0000 7f2e4bfff640 10 librados: watch_flush exit 2024-03-29T11:24:24.843 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.861+0000 7f2e4bfff640 20 client.4388.objecter shutdown clearing up homeless session... 2024-03-29T11:24:24.843 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.861+0000 7f2e4bfff640 10 client.4388.objecter successfully canceled tick 2024-03-29T11:24:24.843 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.862+0000 7f2e4bfff640 1 -- 172.21.15.86:0/2281439370 >> v1:172.21.15.196:6816/588110318 conn(0x7f2e4c06da40 legacy=0x7f2e4c06ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:24.843 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.862+0000 7f2e4bfff640 10 monclient: shutdown 2024-03-29T11:24:24.844 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.862+0000 7f2e4bfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:24.844 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.862+0000 7f2e4bfff640 1 -- 172.21.15.86:0/2281439370 >> v1:172.21.15.86:6790/0 conn(0x7f2e60030660 legacy=0x7f2e6002f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:24.844 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.863+0000 7f2e4bfff640 1 -- 172.21.15.86:0/2281439370 shutdown_connections 2024-03-29T11:24:24.844 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.863+0000 7f2e4bfff640 1 -- 172.21.15.86:0/2281439370 >> 172.21.15.86:0/2281439370 conn(0x7f2e60005c00 msgr2=0x7f2e60034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:24.846 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.864+0000 7f2e4bfff640 1 -- 172.21.15.86:0/2281439370 shutdown_connections 2024-03-29T11:24:24.846 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.864+0000 7f2e4bfff640 1 -- 172.21.15.86:0/2281439370 wait complete. 2024-03-29T11:24:24.846 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:24.865+0000 7f2e4bfff640 1 librados: shutdown 2024-03-29T11:24:24.863 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:31.670 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:31.826 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.841+0000 7f2ec69c5640 10 monclient: get_monmap_and_config 2024-03-29T11:24:31.826 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.841+0000 7f2ec69c5640 10 monclient: build_initial_monmap 2024-03-29T11:24:31.826 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.841+0000 7f2ec69c5640 10 monclient: monmap: 2024-03-29T11:24:31.826 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:24:31.827 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:31.827 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:24:31.842288+0000 2024-03-29T11:24:31.827 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:24:31.842288+0000 2024-03-29T11:24:31.827 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:31.827 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:31.827 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:24:31.827 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:24:31.827 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:24:31.827 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:31.828 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.843+0000 7f2ec69c5640 1 Processor -- start 2024-03-29T11:24:31.830 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.844+0000 7f2ec69c5640 1 -- start start 2024-03-29T11:24:31.830 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.844+0000 7f2ec69c5640 10 monclient: init 2024-03-29T11:24:31.831 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.845+0000 7f2ec69c5640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:31.831 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.845+0000 7f2ec69c5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:31.831 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.845+0000 7f2ec69c5640 10 monclient(hunting): picked mon.noname-b con 0x7f2ec00b2800 addr v1:172.21.15.86:6790/0 2024-03-29T11:24:31.831 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.845+0000 7f2ec69c5640 10 monclient(hunting): picked mon.noname-a con 0x7f2ec00a9c00 addr v1:172.21.15.86:6789/0 2024-03-29T11:24:31.831 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.845+0000 7f2ec69c5640 10 monclient(hunting): picked mon.noname-c con 0x7f2ec00b38c0 addr v1:172.21.15.196:6789/0 2024-03-29T11:24:31.831 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.845+0000 7f2ec69c5640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ec0141eb0 con 0x7f2ec00a9c00 2024-03-29T11:24:31.831 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.845+0000 7f2ec69c5640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ec00b22d0 con 0x7f2ec00b38c0 2024-03-29T11:24:31.831 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.845+0000 7f2ec69c5640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ec0142d30 con 0x7f2ec00b2800 2024-03-29T11:24:31.831 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.845+0000 7f2ec69c5640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:31.831 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.845+0000 7f2ec69c5640 10 monclient(hunting): authenticate will time out at 1096.163495s 2024-03-29T11:24:31.832 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.849+0000 7f2ec61c4640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f2ec00b38c0 0x7f2ec014d7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:53763/0 (socket says 172.21.15.86:53763) 2024-03-29T11:24:31.832 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.849+0000 7f2ec61c4640 1 -- 172.21.15.86:0/1396098515 learned_addr learned my addr 172.21.15.86:0/1396098515 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:24:31.834 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.852+0000 7f2ec49c1640 1 -- 172.21.15.86:0/1396098515 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3629342683 0 0) 0x7f2ec0141eb0 con 0x7f2ec00a9c00 2024-03-29T11:24:31.834 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.852+0000 7f2ec49c1640 10 monclient(hunting): _negotiate 2024-03-29T11:24:31.834 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.852+0000 7f2ec49c1640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:31.834 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.852+0000 7f2ec49c1640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:31.834 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.852+0000 7f2ec49c1640 10 monclient(hunting): my global_id is 4374 2024-03-29T11:24:31.834 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.852+0000 7f2ec49c1640 1 -- 172.21.15.86:0/1396098515 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2eac003520 con 0x7f2ec00a9c00 2024-03-29T11:24:31.834 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.852+0000 7f2ec49c1640 1 -- 172.21.15.86:0/1396098515 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2005138120 0 0) 0x7f2ec00b22d0 con 0x7f2ec00b38c0 2024-03-29T11:24:31.835 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.852+0000 7f2ec49c1640 10 monclient(hunting): _negotiate 2024-03-29T11:24:31.835 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.852+0000 7f2ec49c1640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:31.835 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.852+0000 7f2ec49c1640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:31.835 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.852+0000 7f2ec49c1640 10 monclient(hunting): my global_id is 4336 2024-03-29T11:24:31.835 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.852+0000 7f2ec49c1640 1 -- 172.21.15.86:0/1396098515 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ec0141eb0 con 0x7f2ec00b38c0 2024-03-29T11:24:31.836 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.853+0000 7f2ec49c1640 1 -- 172.21.15.86:0/1396098515 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4664387 0 0) 0x7f2eac003520 con 0x7f2ec00a9c00 2024-03-29T11:24:31.836 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.853+0000 7f2ec49c1640 1 -- 172.21.15.86:0/1396098515 >> v1:172.21.15.86:6790/0 conn(0x7f2ec00b2800 legacy=0x7f2ec0142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:31.836 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.853+0000 7f2ec49c1640 1 -- 172.21.15.86:0/1396098515 >> v1:172.21.15.196:6789/0 conn(0x7f2ec00b38c0 legacy=0x7f2ec014d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:31.836 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.853+0000 7f2ec49c1640 10 monclient: _finish_hunting 0 2024-03-29T11:24:31.836 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.853+0000 7f2ec49c1640 1 monclient: found mon.noname-a 2024-03-29T11:24:31.837 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.853+0000 7f2ec49c1640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:31.837 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.853+0000 7f2ec49c1640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.86:6789/0 2024-03-29T11:24:31.837 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.853+0000 7f2ec49c1640 1 -- 172.21.15.86:0/1396098515 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec00aaf90 con 0x7f2ec00a9c00 2024-03-29T11:24:31.837 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.853+0000 7f2ec49c1640 10 monclient: _finish_auth 0 2024-03-29T11:24:31.837 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.853+0000 7f2ec49c1640 10 monclient: _check_auth_tickets 2024-03-29T11:24:31.837 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.853+0000 7f2ec49c1640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:31.837 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.853+0000 7f2ec49c1640 1 -- 172.21.15.86:0/1396098515 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f2eb0001690 con 0x7f2ec00a9c00 2024-03-29T11:24:31.837 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.853+0000 7f2ec49c1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:31.837 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.853+0000 7f2ec49c1640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:31.837 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.853+0000 7f2ec49c1640 10 monclient: dump: 2024-03-29T11:24:31.837 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:24:31.837 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:31.837 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:31.837 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:31.837 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:31.838 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:31.838 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:24:31.838 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:24:31.838 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:24:31.838 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:31.838 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.853+0000 7f2ec69c5640 5 monclient: authenticate success, global_id 4374 2024-03-29T11:24:31.838 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.853+0000 7f2ec69c5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:31.838 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.853+0000 7f2ec49c1640 1 -- 172.21.15.86:0/1396098515 <== mon.0 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2eb00017f0 con 0x7f2ec00a9c00 2024-03-29T11:24:31.838 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.853+0000 7f2ec49c1640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:31.838 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.853+0000 7f2ec49c1640 1 -- 172.21.15.86:0/1396098515 <== mon.0 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f2eb0001bc0 con 0x7f2ec00a9c00 2024-03-29T11:24:31.838 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.853+0000 7f2ec49c1640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:31.838 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.853+0000 7f2ec49c1640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:31.838 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.853+0000 7f2ec49c1640 10 monclient: dump: 2024-03-29T11:24:31.838 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:24:31.839 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:31.839 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:31.839 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:31.839 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:31.839 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:31.839 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:24:31.839 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:24:31.839 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:24:31.839 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:31.839 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.854+0000 7f2ec69c5640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:31.839 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.854+0000 7f2ec69c5640 4 set_mon_vals no callback set 2024-03-29T11:24:31.840 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.855+0000 7f2ec69c5640 10 monclient: shutdown 2024-03-29T11:24:31.840 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.855+0000 7f2ec69c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:31.840 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.855+0000 7f2ec69c5640 1 -- 172.21.15.86:0/1396098515 >> v1:172.21.15.86:6789/0 conn(0x7f2ec00a9c00 legacy=0x7f2ec013bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:31.840 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.855+0000 7f2ec69c5640 1 -- 172.21.15.86:0/1396098515 shutdown_connections 2024-03-29T11:24:31.840 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.855+0000 7f2ec69c5640 1 -- 172.21.15.86:0/1396098515 >> 172.21.15.86:0/1396098515 conn(0x7f2ec0066a00 msgr2=0x7f2ec0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:31.840 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.857+0000 7f2ec69c5640 1 -- 172.21.15.86:0/1396098515 shutdown_connections 2024-03-29T11:24:31.840 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.858+0000 7f2ec69c5640 1 -- 172.21.15.86:0/1396098515 wait complete. 2024-03-29T11:24:31.841 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.859+0000 7f2ec69c5640 10 monclient: build_initial_monmap 2024-03-29T11:24:31.841 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.859+0000 7f2ec69c5640 10 monclient: monmap: 2024-03-29T11:24:31.841 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:24:31.841 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:31.842 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:24:31.842 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:24:31.842 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:31.842 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:31.842 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:24:31.842 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:24:31.842 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:24:31.842 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:31.842 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.859+0000 7f2ec69c5640 1 librados: starting msgr at 2024-03-29T11:24:31.842 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.859+0000 7f2ec69c5640 1 librados: starting objecter 2024-03-29T11:24:31.842 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.859+0000 7f2ec69c5640 1 Processor -- start 2024-03-29T11:24:31.844 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.860+0000 7f2ec69c5640 1 -- start start 2024-03-29T11:24:31.844 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.860+0000 7f2ec69c5640 1 librados: setting wanted keys 2024-03-29T11:24:31.844 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.860+0000 7f2ec69c5640 1 librados: calling monclient init 2024-03-29T11:24:31.844 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.860+0000 7f2ec69c5640 10 monclient: init 2024-03-29T11:24:31.844 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.860+0000 7f2ec69c5640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:31.844 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.860+0000 7f2ec69c5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:31.844 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.860+0000 7f2ec69c5640 10 monclient(hunting): picked mon.noname-a con 0x7f2ec00a9c00 addr v1:172.21.15.86:6789/0 2024-03-29T11:24:31.844 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.860+0000 7f2ec69c5640 10 monclient(hunting): picked mon.noname-b con 0x7f2ec00b38c0 addr v1:172.21.15.196:6789/0 2024-03-29T11:24:31.844 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.860+0000 7f2ec69c5640 10 monclient(hunting): picked mon.noname-c con 0x7f2ec006eca0 addr v1:172.21.15.86:6790/0 2024-03-29T11:24:31.844 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.860+0000 7f2ec69c5640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ec013bdc0 con 0x7f2ec00a9c00 2024-03-29T11:24:31.844 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.860+0000 7f2ec69c5640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ec006fa90 con 0x7f2ec00b38c0 2024-03-29T11:24:31.845 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.860+0000 7f2ec69c5640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ec006fd70 con 0x7f2ec006eca0 2024-03-29T11:24:31.845 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.860+0000 7f2ec69c5640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:31.845 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.860+0000 7f2ec69c5640 10 monclient(hunting): authenticate will time out at 1096.178947s 2024-03-29T11:24:31.849 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.868+0000 7f2ec51c2640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f2ec00b38c0 0x7f2ec0139bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:53873/0 (socket says 172.21.15.86:53873) 2024-03-29T11:24:31.850 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.868+0000 7f2ec61c4640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f2ec006eca0 0x7f2ec01ee470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:52479/0 (socket says 172.21.15.86:52479) 2024-03-29T11:24:31.850 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.869+0000 7f2ec61c4640 1 -- 172.21.15.86:0/2711007486 learned_addr learned my addr 172.21.15.86:0/2711007486 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:24:31.855 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.870+0000 7f2eb67fc640 10 client.?.objecter ms_handle_connect 0x7f2ec006eca0 2024-03-29T11:24:31.855 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.870+0000 7f2eb67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:31.855 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.870+0000 7f2eb67fc640 10 client.?.objecter ms_handle_connect 0x7f2ec00b38c0 2024-03-29T11:24:31.855 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.870+0000 7f2eb67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:31.855 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.870+0000 7f2eb67fc640 1 -- 172.21.15.86:0/2711007486 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3091324944 0 0) 0x7f2ec006fd70 con 0x7f2ec006eca0 2024-03-29T11:24:31.855 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.870+0000 7f2eb67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:31.855 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.870+0000 7f2eb67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:31.855 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.870+0000 7f2eb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:31.855 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.870+0000 7f2eb67fc640 10 monclient(hunting): my global_id is 4391 2024-03-29T11:24:31.855 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.870+0000 7f2eb67fc640 1 -- 172.21.15.86:0/2711007486 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ea4004eb0 con 0x7f2ec006eca0 2024-03-29T11:24:31.856 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.873+0000 7f2eb67fc640 1 -- 172.21.15.86:0/2711007486 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1719662587 0 0) 0x7f2ec006fa90 con 0x7f2ec00b38c0 2024-03-29T11:24:31.856 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.873+0000 7f2eb67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:31.856 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.873+0000 7f2eb67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:31.856 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.873+0000 7f2eb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:31.856 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.873+0000 7f2eb67fc640 10 monclient(hunting): my global_id is 4339 2024-03-29T11:24:31.856 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.873+0000 7f2eb67fc640 1 -- 172.21.15.86:0/2711007486 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ec006fd70 con 0x7f2ec00b38c0 2024-03-29T11:24:31.856 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.873+0000 7f2eb67fc640 10 client.?.objecter ms_handle_connect 0x7f2ec00a9c00 2024-03-29T11:24:31.856 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.873+0000 7f2eb67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:31.856 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.873+0000 7f2eb67fc640 1 -- 172.21.15.86:0/2711007486 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 40578999 0 0) 0x7f2ec006fd70 con 0x7f2ec00b38c0 2024-03-29T11:24:31.856 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.873+0000 7f2eb67fc640 1 -- 172.21.15.86:0/2711007486 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ea4006540 con 0x7f2ec00b38c0 2024-03-29T11:24:31.856 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.873+0000 7f2eb67fc640 1 -- 172.21.15.86:0/2711007486 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f2eb0001b00 con 0x7f2ec00b38c0 2024-03-29T11:24:31.856 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.873+0000 7f2eb67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:31.856 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.873+0000 7f2eb67fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:31.856 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.873+0000 7f2eb67fc640 10 monclient(hunting): dump: 2024-03-29T11:24:31.857 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:24:31.857 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:31.857 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:31.857 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:31.857 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:31.857 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:31.857 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:24:31.857 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:24:31.857 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:24:31.857 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:31.857 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.873+0000 7f2eb67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:31.857 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.873+0000 7f2eb67fc640 1 -- 172.21.15.86:0/2711007486 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3418053995 0 0) 0x7f2ec013bdc0 con 0x7f2ec00a9c00 2024-03-29T11:24:31.857 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.873+0000 7f2eb67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:31.857 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.873+0000 7f2eb67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:31.857 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.873+0000 7f2eb67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:31.858 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.873+0000 7f2eb67fc640 10 monclient(hunting): my global_id is 4377 2024-03-29T11:24:31.858 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.873+0000 7f2eb67fc640 1 -- 172.21.15.86:0/2711007486 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ea4006a70 con 0x7f2ec00a9c00 2024-03-29T11:24:31.859 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.874+0000 7f2eb67fc640 1 -- 172.21.15.86:0/2711007486 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3370823947 0 0) 0x7f2ea4006540 con 0x7f2ec00b38c0 2024-03-29T11:24:31.859 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.874+0000 7f2eb67fc640 1 -- 172.21.15.86:0/2711007486 >> v1:172.21.15.86:6790/0 conn(0x7f2ec006eca0 legacy=0x7f2ec01ee470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:31.859 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.874+0000 7f2eb67fc640 1 -- 172.21.15.86:0/2711007486 >> v1:172.21.15.86:6789/0 conn(0x7f2ec00a9c00 legacy=0x7f2ec01394b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:31.859 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.874+0000 7f2eb67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:31.859 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.874+0000 7f2eb67fc640 1 monclient: found mon.b 2024-03-29T11:24:31.859 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.874+0000 7f2eb67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:31.859 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.874+0000 7f2eb67fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:24:31.859 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.874+0000 7f2eb67fc640 1 -- 172.21.15.86:0/2711007486 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ec0070050 con 0x7f2ec00b38c0 2024-03-29T11:24:31.859 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.874+0000 7f2eb67fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:31.859 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.874+0000 7f2eb67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:31.859 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.874+0000 7f2eb67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:31.859 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.877+0000 7f2ec69c5640 5 monclient: authenticate success, global_id 4339 2024-03-29T11:24:31.859 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.877+0000 7f2ec69c5640 10 monclient: _renew_subs 2024-03-29T11:24:31.860 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.877+0000 7f2ec69c5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:24:31.860 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.877+0000 7f2ec69c5640 1 -- 172.21.15.86:0/2711007486 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ec006e100 con 0x7f2ec00b38c0 2024-03-29T11:24:31.860 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.877+0000 7f2ec69c5640 10 client.4339.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:31.861 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.877+0000 7f2ec69c5640 10 monclient: _renew_subs 2024-03-29T11:24:31.861 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.877+0000 7f2ec69c5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:24:31.861 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.877+0000 7f2ec69c5640 1 -- 172.21.15.86:0/2711007486 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ec01efb90 con 0x7f2ec00b38c0 2024-03-29T11:24:31.861 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.877+0000 7f2ec69c5640 1 librados: init done 2024-03-29T11:24:31.861 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.877+0000 7f2eb67fc640 1 -- 172.21.15.86:0/2711007486 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2eb00038a0 con 0x7f2ec00b38c0 2024-03-29T11:24:31.861 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.877+0000 7f2eb67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:31.861 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.877+0000 7f2eb67fc640 1 -- 172.21.15.86:0/2711007486 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f2eb0001b00 con 0x7f2ec00b38c0 2024-03-29T11:24:31.861 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.877+0000 7f2eb67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:31.861 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.877+0000 7f2eb67fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:31.861 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.877+0000 7f2eb7fff640 4 set_mon_vals no callback set 2024-03-29T11:24:31.861 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.877+0000 7f2eb67fc640 10 monclient: dump: 2024-03-29T11:24:31.861 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:24:31.861 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:31.862 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:31.862 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:31.862 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:31.862 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:31.862 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:24:31.862 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:24:31.862 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:24:31.862 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:31.862 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.878+0000 7f2ec69c5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:31.862 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.879+0000 7f2ec69c5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:31.862 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.879+0000 7f2ec69c5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:24:31.862 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.879+0000 7f2ec69c5640 1 -- 172.21.15.86:0/2711007486 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ea8005080 con 0x7f2ec00b38c0 2024-03-29T11:24:31.864 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.882+0000 7f2eb67fc640 1 -- 172.21.15.86:0/2711007486 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f2eb0003b50 con 0x7f2ec00b38c0 2024-03-29T11:24:31.865 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.883+0000 7f2eb67fc640 1 -- 172.21.15.86:0/2711007486 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f2eb00545e0 con 0x7f2ec00b38c0 2024-03-29T11:24:31.865 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.883+0000 7f2eb67fc640 10 client.4339.objecter ms_dispatch 0x7f2ec0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:31.865 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.883+0000 7f2eb67fc640 3 client.4339.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:31.865 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.883+0000 7f2eb67fc640 3 client.4339.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:31.865 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.883+0000 7f2eb67fc640 20 client.4339.objecter dump_active .. 0 homeless 2024-03-29T11:24:31.868 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.886+0000 7f2ec59c3640 10 monclient: get_auth_request con 0x7f2ea406dfe0 auth_method 0 2024-03-29T11:24:31.868 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.887+0000 7f2eb67fc640 10 client.4339.objecter ms_handle_connect 0x7f2ea406dfe0 2024-03-29T11:24:31.875 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.893+0000 7f2eb67fc640 1 -- 172.21.15.86:0/2711007486 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2eb0050880 con 0x7f2ec00b38c0 2024-03-29T11:24:31.875 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.893+0000 7f2eb67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:31.875 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:31.893+0000 7f2eb67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:32.203 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:32.221+0000 7f2ec69c5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:32.204 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:32.221+0000 7f2ec69c5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:32.204 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:32.221+0000 7f2ec69c5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:24:32.204 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:32.221+0000 7f2ec69c5640 1 -- 172.21.15.86:0/2711007486 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ea8005330 con 0x7f2ec00b38c0 2024-03-29T11:24:32.204 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:32.222+0000 7f2eb67fc640 1 -- 172.21.15.86:0/2711007486 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f2eb0050880 con 0x7f2ec00b38c0 2024-03-29T11:24:32.204 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:32.222+0000 7f2eb67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:32.204 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:32.222+0000 7f2eb67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:32.205 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:24:32.206 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:32.210 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:32.227+0000 7f2ec69c5640 10 librados: watch_flush enter 2024-03-29T11:24:32.210 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:32.227+0000 7f2ec69c5640 10 librados: watch_flush exit 2024-03-29T11:24:32.210 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:32.227+0000 7f2ec69c5640 20 client.4339.objecter shutdown clearing up homeless session... 2024-03-29T11:24:32.210 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:32.227+0000 7f2ec69c5640 10 client.4339.objecter successfully canceled tick 2024-03-29T11:24:32.210 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:32.227+0000 7f2ec69c5640 1 -- 172.21.15.86:0/2711007486 >> v1:172.21.15.196:6816/588110318 conn(0x7f2ea406dfe0 legacy=0x7f2ea40704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:32.210 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:32.227+0000 7f2ec69c5640 10 monclient: shutdown 2024-03-29T11:24:32.210 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:32.227+0000 7f2ec69c5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:32.210 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:32.227+0000 7f2ec69c5640 1 -- 172.21.15.86:0/2711007486 >> v1:172.21.15.196:6789/0 conn(0x7f2ec00b38c0 legacy=0x7f2ec0139bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:32.210 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:32.228+0000 7f2ec69c5640 1 -- 172.21.15.86:0/2711007486 shutdown_connections 2024-03-29T11:24:32.210 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:32.228+0000 7f2ec69c5640 1 -- 172.21.15.86:0/2711007486 >> 172.21.15.86:0/2711007486 conn(0x7f2ec0066a00 msgr2=0x7f2ec0145240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:32.210 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:32.228+0000 7f2ec69c5640 1 -- 172.21.15.86:0/2711007486 shutdown_connections 2024-03-29T11:24:32.210 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:32.228+0000 7f2ec69c5640 1 -- 172.21.15.86:0/2711007486 wait complete. 2024-03-29T11:24:32.210 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:32.228+0000 7f2ec69c5640 1 librados: shutdown 2024-03-29T11:24:32.227 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:39.033 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:39.184 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.202+0000 7fd00103b640 10 monclient: get_monmap_and_config 2024-03-29T11:24:39.184 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.202+0000 7fd00103b640 10 monclient: build_initial_monmap 2024-03-29T11:24:39.184 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.202+0000 7fd00103b640 10 monclient: monmap: 2024-03-29T11:24:39.184 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:24:39.184 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:39.184 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:24:39.203150+0000 2024-03-29T11:24:39.184 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:24:39.203150+0000 2024-03-29T11:24:39.184 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:39.185 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:39.185 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:24:39.185 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:24:39.185 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:24:39.185 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:39.186 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.203+0000 7fd00103b640 1 Processor -- start 2024-03-29T11:24:39.186 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.203+0000 7fd00103b640 1 -- start start 2024-03-29T11:24:39.186 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.203+0000 7fd00103b640 10 monclient: init 2024-03-29T11:24:39.187 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.203+0000 7fd00103b640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:39.187 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.203+0000 7fd00103b640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:24:39.187 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.204+0000 7fd00103b640 10 monclient(hunting): picked mon.noname-a con 0x7fcffc1457f0 addr v1:172.21.15.86:6789/0 2024-03-29T11:24:39.187 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.204+0000 7fd00103b640 10 monclient(hunting): picked mon.noname-c con 0x7fcffc13f650 addr v1:172.21.15.196:6789/0 2024-03-29T11:24:39.187 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.204+0000 7fd00103b640 10 monclient(hunting): picked mon.noname-b con 0x7fcffc0a9d70 addr v1:172.21.15.86:6790/0 2024-03-29T11:24:39.187 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.204+0000 7fd00103b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcffc0aaef0 con 0x7fcffc1457f0 2024-03-29T11:24:39.187 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.204+0000 7fd00103b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcffc145bd0 con 0x7fcffc13f650 2024-03-29T11:24:39.187 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.204+0000 7fd00103b640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcffc145eb0 con 0x7fcffc0a9d70 2024-03-29T11:24:39.187 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.204+0000 7fd00103b640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:39.187 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.204+0000 7fd00103b640 10 monclient(hunting): authenticate will time out at 1103.522106s 2024-03-29T11:24:39.189 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.206+0000 7fcffad9d640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fcffc1457f0 0x7fcffc13ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:49741/0 (socket says 172.21.15.86:49741) 2024-03-29T11:24:39.189 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.206+0000 7fcffad9d640 1 -- 172.21.15.86:0/1635062443 learned_addr learned my addr 172.21.15.86:0/1635062443 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:24:39.194 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.211+0000 7fcff9d9b640 1 -- 172.21.15.86:0/1635062443 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 794278760 0 0) 0x7fcffc0aaef0 con 0x7fcffc1457f0 2024-03-29T11:24:39.194 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.211+0000 7fcff9d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:24:39.194 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.211+0000 7fcff9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:39.194 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.211+0000 7fcff9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:39.194 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.211+0000 7fcff9d9b640 10 monclient(hunting): my global_id is 4380 2024-03-29T11:24:39.194 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.211+0000 7fcff9d9b640 1 -- 172.21.15.86:0/1635062443 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfe4003520 con 0x7fcffc1457f0 2024-03-29T11:24:39.194 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.211+0000 7fcff9d9b640 1 -- 172.21.15.86:0/1635062443 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 49182649 0 0) 0x7fcffc145eb0 con 0x7fcffc0a9d70 2024-03-29T11:24:39.195 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.211+0000 7fcff9d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:24:39.195 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.211+0000 7fcff9d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:39.195 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.212+0000 7fcff9d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:39.195 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.212+0000 7fcff9d9b640 10 monclient(hunting): my global_id is 4394 2024-03-29T11:24:39.195 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.212+0000 7fcff9d9b640 1 -- 172.21.15.86:0/1635062443 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcffc0aaef0 con 0x7fcffc0a9d70 2024-03-29T11:24:39.195 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.212+0000 7fcff9d9b640 1 -- 172.21.15.86:0/1635062443 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1290066809 0 0) 0x7fcfe4003520 con 0x7fcffc1457f0 2024-03-29T11:24:39.195 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.212+0000 7fcff9d9b640 1 -- 172.21.15.86:0/1635062443 >> v1:172.21.15.86:6790/0 conn(0x7fcffc0a9d70 legacy=0x7fcffc14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:39.195 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.212+0000 7fcff9d9b640 1 -- 172.21.15.86:0/1635062443 >> v1:172.21.15.196:6789/0 conn(0x7fcffc13f650 legacy=0x7fcffc0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:39.196 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.212+0000 7fcff9d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:24:39.196 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.212+0000 7fcff9d9b640 1 monclient: found mon.noname-a 2024-03-29T11:24:39.196 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.212+0000 7fcff9d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:39.196 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.212+0000 7fcff9d9b640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.86:6789/0 2024-03-29T11:24:39.196 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.212+0000 7fcff9d9b640 1 -- 172.21.15.86:0/1635062443 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcffc146340 con 0x7fcffc1457f0 2024-03-29T11:24:39.196 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.212+0000 7fcff9d9b640 10 monclient: _finish_auth 0 2024-03-29T11:24:39.196 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.212+0000 7fcff9d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:24:39.196 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.212+0000 7fcff9d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:39.196 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.212+0000 7fcff9d9b640 1 -- 172.21.15.86:0/1635062443 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fcfec001690 con 0x7fcffc1457f0 2024-03-29T11:24:39.196 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.212+0000 7fcff9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:39.196 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.212+0000 7fcff9d9b640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:24:39.196 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.212+0000 7fcff9d9b640 10 monclient: dump: 2024-03-29T11:24:39.196 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:24:39.196 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:39.197 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:39.197 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:39.197 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:39.197 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:39.197 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:24:39.197 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:24:39.197 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:24:39.197 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:39.197 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.213+0000 7fcff9d9b640 1 -- 172.21.15.86:0/1635062443 <== mon.0 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcfec0018b0 con 0x7fcffc1457f0 2024-03-29T11:24:39.197 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.213+0000 7fcff9d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:39.197 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.213+0000 7fcff9d9b640 1 -- 172.21.15.86:0/1635062443 <== mon.0 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fcfec001c40 con 0x7fcffc1457f0 2024-03-29T11:24:39.197 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.213+0000 7fcff9d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:39.197 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.213+0000 7fcff9d9b640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:24:39.197 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.213+0000 7fcff9d9b640 10 monclient: dump: 2024-03-29T11:24:39.197 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:24:39.198 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:39.198 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:39.198 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:39.198 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:39.198 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:39.198 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:24:39.198 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:24:39.198 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:24:39.198 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:39.199 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.214+0000 7fd00103b640 5 monclient: authenticate success, global_id 4380 2024-03-29T11:24:39.199 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.214+0000 7fd00103b640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:39.200 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.214+0000 7fd00103b640 4 set_mon_vals no callback set 2024-03-29T11:24:39.200 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.215+0000 7fd00103b640 10 monclient: shutdown 2024-03-29T11:24:39.200 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.215+0000 7fd00103b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:39.200 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.215+0000 7fd00103b640 1 -- 172.21.15.86:0/1635062443 >> v1:172.21.15.86:6789/0 conn(0x7fcffc1457f0 legacy=0x7fcffc13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:39.200 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.216+0000 7fd00103b640 1 -- 172.21.15.86:0/1635062443 shutdown_connections 2024-03-29T11:24:39.200 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.216+0000 7fd00103b640 1 -- 172.21.15.86:0/1635062443 >> 172.21.15.86:0/1635062443 conn(0x7fcffc066a00 msgr2=0x7fcffc066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:39.200 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.216+0000 7fd00103b640 1 -- 172.21.15.86:0/1635062443 shutdown_connections 2024-03-29T11:24:39.200 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.216+0000 7fd00103b640 1 -- 172.21.15.86:0/1635062443 wait complete. 2024-03-29T11:24:39.200 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.217+0000 7fd00103b640 10 monclient: build_initial_monmap 2024-03-29T11:24:39.200 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.217+0000 7fd00103b640 10 monclient: monmap: 2024-03-29T11:24:39.200 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:24:39.200 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:39.200 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:24:39.200 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:24:39.201 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:39.201 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:39.201 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:24:39.201 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:24:39.201 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:24:39.201 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:39.201 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.218+0000 7fd00103b640 1 librados: starting msgr at 2024-03-29T11:24:39.202 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.218+0000 7fd00103b640 1 librados: starting objecter 2024-03-29T11:24:39.202 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.218+0000 7fd00103b640 1 Processor -- start 2024-03-29T11:24:39.202 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.218+0000 7fd00103b640 1 -- start start 2024-03-29T11:24:39.202 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.218+0000 7fd00103b640 1 librados: setting wanted keys 2024-03-29T11:24:39.202 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.218+0000 7fd00103b640 1 librados: calling monclient init 2024-03-29T11:24:39.202 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.218+0000 7fd00103b640 10 monclient: init 2024-03-29T11:24:39.202 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.219+0000 7fd00103b640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:39.202 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.219+0000 7fd00103b640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:39.202 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.219+0000 7fd00103b640 10 monclient(hunting): picked mon.noname-b con 0x7fcffc0a9d70 addr v1:172.21.15.196:6789/0 2024-03-29T11:24:39.202 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.219+0000 7fd00103b640 10 monclient(hunting): picked mon.noname-a con 0x7fcffc13f650 addr v1:172.21.15.86:6789/0 2024-03-29T11:24:39.202 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.219+0000 7fd00103b640 10 monclient(hunting): picked mon.noname-c con 0x7fcffc1457f0 addr v1:172.21.15.86:6790/0 2024-03-29T11:24:39.202 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.219+0000 7fd00103b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcffc1421e0 con 0x7fcffc13f650 2024-03-29T11:24:39.202 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.219+0000 7fd00103b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcffc143820 con 0x7fcffc0a9d70 2024-03-29T11:24:39.202 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.219+0000 7fd00103b640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcffc1439b0 con 0x7fcffc1457f0 2024-03-29T11:24:39.203 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.219+0000 7fd00103b640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:39.203 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.219+0000 7fd00103b640 10 monclient(hunting): authenticate will time out at 1103.537370s 2024-03-29T11:24:39.204 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.222+0000 7fcffb59e640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7fcffc1457f0 0x7fcffc143e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:41947/0 (socket says 172.21.15.86:41947) 2024-03-29T11:24:39.204 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.222+0000 7fcffb59e640 1 -- 172.21.15.86:0/2639566565 learned_addr learned my addr 172.21.15.86:0/2639566565 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:24:39.204 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.222+0000 7fcfe37fe640 10 client.?.objecter ms_handle_connect 0x7fcffc1457f0 2024-03-29T11:24:39.204 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.222+0000 7fcfe37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:39.211 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.222+0000 7fcfe37fe640 1 -- 172.21.15.86:0/2639566565 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2871685380 0 0) 0x7fcffc1439b0 con 0x7fcffc1457f0 2024-03-29T11:24:39.211 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.224+0000 7fcfe37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:39.211 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.224+0000 7fcfe37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:39.211 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.224+0000 7fcfe37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:39.211 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.224+0000 7fcfe37fe640 10 monclient(hunting): my global_id is 4397 2024-03-29T11:24:39.211 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.224+0000 7fcfe37fe640 1 -- 172.21.15.86:0/2639566565 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcfe8004eb0 con 0x7fcffc1457f0 2024-03-29T11:24:39.211 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.225+0000 7fcfe37fe640 10 client.?.objecter ms_handle_connect 0x7fcffc0a9d70 2024-03-29T11:24:39.212 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.225+0000 7fcfe37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:39.212 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.225+0000 7fcfe37fe640 1 -- 172.21.15.86:0/2639566565 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3244624713 0 0) 0x7fcffc143820 con 0x7fcffc0a9d70 2024-03-29T11:24:39.212 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.225+0000 7fcfe37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:39.212 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.225+0000 7fcfe37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:39.212 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.225+0000 7fcfe37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:39.212 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.225+0000 7fcfe37fe640 10 monclient(hunting): my global_id is 4342 2024-03-29T11:24:39.212 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.225+0000 7fcfe37fe640 1 -- 172.21.15.86:0/2639566565 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcffc1439b0 con 0x7fcffc0a9d70 2024-03-29T11:24:39.212 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.225+0000 7fcfe37fe640 10 client.?.objecter ms_handle_connect 0x7fcffc13f650 2024-03-29T11:24:39.212 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.226+0000 7fcfe37fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:39.212 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.226+0000 7fcfe37fe640 1 -- 172.21.15.86:0/2639566565 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1369552279 0 0) 0x7fcfe8004eb0 con 0x7fcffc1457f0 2024-03-29T11:24:39.212 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.226+0000 7fcfe37fe640 1 -- 172.21.15.86:0/2639566565 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcffc143820 con 0x7fcffc1457f0 2024-03-29T11:24:39.212 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.226+0000 7fcfe37fe640 1 -- 172.21.15.86:0/2639566565 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fcff0001940 con 0x7fcffc1457f0 2024-03-29T11:24:39.213 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.226+0000 7fcfe37fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:39.213 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.226+0000 7fcfe37fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:39.213 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.226+0000 7fcfe37fe640 10 monclient(hunting): dump: 2024-03-29T11:24:39.213 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:24:39.213 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:39.213 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:39.213 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:39.213 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:39.213 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:39.213 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:24:39.213 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:24:39.213 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:24:39.213 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:39.213 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.226+0000 7fcfe37fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:39.213 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.226+0000 7fcfe37fe640 1 -- 172.21.15.86:0/2639566565 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 729741802 0 0) 0x7fcffc1439b0 con 0x7fcffc0a9d70 2024-03-29T11:24:39.214 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.226+0000 7fcfe37fe640 1 -- 172.21.15.86:0/2639566565 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcfe8004eb0 con 0x7fcffc0a9d70 2024-03-29T11:24:39.214 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.226+0000 7fcfe37fe640 1 -- 172.21.15.86:0/2639566565 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2165350762 0 0) 0x7fcffc1421e0 con 0x7fcffc13f650 2024-03-29T11:24:39.214 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.226+0000 7fcfe37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:24:39.214 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.226+0000 7fcfe37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:39.214 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.226+0000 7fcfe37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:39.214 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.227+0000 7fcfe37fe640 10 monclient(hunting): my global_id is 4383 2024-03-29T11:24:39.214 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.227+0000 7fcfe37fe640 1 -- 172.21.15.86:0/2639566565 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcffc1439b0 con 0x7fcffc13f650 2024-03-29T11:24:39.214 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.227+0000 7fcfe37fe640 1 -- 172.21.15.86:0/2639566565 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1869162085 0 0) 0x7fcffc143820 con 0x7fcffc1457f0 2024-03-29T11:24:39.214 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.227+0000 7fcfe37fe640 1 -- 172.21.15.86:0/2639566565 >> v1:172.21.15.196:6789/0 conn(0x7fcffc0a9d70 legacy=0x7fcffc140010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:39.214 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.227+0000 7fcfe37fe640 1 -- 172.21.15.86:0/2639566565 >> v1:172.21.15.86:6789/0 conn(0x7fcffc13f650 legacy=0x7fcffc143110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:39.214 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.227+0000 7fcfe37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:24:39.214 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.227+0000 7fcfe37fe640 1 monclient: found mon.c 2024-03-29T11:24:39.214 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.227+0000 7fcfe37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:39.214 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.227+0000 7fcfe37fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:24:39.215 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.227+0000 7fcfe37fe640 1 -- 172.21.15.86:0/2639566565 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcffc144510 con 0x7fcffc1457f0 2024-03-29T11:24:39.215 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.227+0000 7fcfe37fe640 10 monclient: _finish_auth 0 2024-03-29T11:24:39.215 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.227+0000 7fcfe37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:24:39.215 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.227+0000 7fcfe37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:39.215 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.227+0000 7fd00103b640 5 monclient: authenticate success, global_id 4397 2024-03-29T11:24:39.215 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.227+0000 7fd00103b640 10 monclient: _renew_subs 2024-03-29T11:24:39.215 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.227+0000 7fd00103b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:24:39.215 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.227+0000 7fd00103b640 1 -- 172.21.15.86:0/2639566565 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcffc144760 con 0x7fcffc1457f0 2024-03-29T11:24:39.215 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.227+0000 7fd00103b640 10 client.4397.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:39.215 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.227+0000 7fd00103b640 10 monclient: _renew_subs 2024-03-29T11:24:39.215 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.227+0000 7fd00103b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:24:39.215 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.227+0000 7fd00103b640 1 -- 172.21.15.86:0/2639566565 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcffc140be0 con 0x7fcffc1457f0 2024-03-29T11:24:39.215 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.227+0000 7fd00103b640 1 librados: init done 2024-03-29T11:24:39.215 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.227+0000 7fcfe37fe640 1 -- 172.21.15.86:0/2639566565 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcff0001aa0 con 0x7fcffc1457f0 2024-03-29T11:24:39.215 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.227+0000 7fcfe37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:39.216 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.227+0000 7fcfe37fe640 1 -- 172.21.15.86:0/2639566565 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fcff00038c0 con 0x7fcffc1457f0 2024-03-29T11:24:39.216 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.227+0000 7fcfe37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:39.216 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.227+0000 7fcfe37fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:24:39.216 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.227+0000 7fcfe37fe640 10 monclient: dump: 2024-03-29T11:24:39.216 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:24:39.216 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:39.216 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:39.216 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:39.216 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:39.216 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:39.216 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:24:39.216 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:24:39.216 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:24:39.216 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:39.216 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.227+0000 7fcff8d99640 4 set_mon_vals no callback set 2024-03-29T11:24:39.218 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.231+0000 7fd00103b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:39.218 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.231+0000 7fd00103b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:39.218 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.231+0000 7fd00103b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:24:39.218 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.231+0000 7fd00103b640 1 -- 172.21.15.86:0/2639566565 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfdc005080 con 0x7fcffc1457f0 2024-03-29T11:24:39.218 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.232+0000 7fcfe37fe640 1 -- 172.21.15.86:0/2639566565 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7fcff0003a20 con 0x7fcffc1457f0 2024-03-29T11:24:39.218 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.233+0000 7fcfe37fe640 1 -- 172.21.15.86:0/2639566565 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7fcff00849a0 con 0x7fcffc1457f0 2024-03-29T11:24:39.218 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.233+0000 7fcfe37fe640 10 client.4397.objecter ms_dispatch 0x7fcffc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:39.218 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.233+0000 7fcfe37fe640 3 client.4397.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:39.218 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.233+0000 7fcfe37fe640 3 client.4397.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:39.219 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.233+0000 7fcffad9d640 10 monclient: get_auth_request con 0x7fcfe806df30 auth_method 0 2024-03-29T11:24:39.219 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.233+0000 7fcfe37fe640 20 client.4397.objecter dump_active .. 0 homeless 2024-03-29T11:24:39.219 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.233+0000 7fcfe37fe640 10 client.4397.objecter ms_handle_connect 0x7fcfe806df30 2024-03-29T11:24:39.224 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.242+0000 7fcfe37fe640 1 -- 172.21.15.86:0/2639566565 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcff0050280 con 0x7fcffc1457f0 2024-03-29T11:24:39.224 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.242+0000 7fcfe37fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:39.224 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.242+0000 7fcfe37fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:39.540 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.558+0000 7fd00103b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:39.540 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.558+0000 7fd00103b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:39.540 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.558+0000 7fd00103b640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:24:39.540 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.558+0000 7fd00103b640 1 -- 172.21.15.86:0/2639566565 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcfdc005330 con 0x7fcffc1457f0 2024-03-29T11:24:39.541 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.559+0000 7fcfe37fe640 1 -- 172.21.15.86:0/2639566565 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7fcff00521b0 con 0x7fcffc1457f0 2024-03-29T11:24:39.541 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.559+0000 7fcfe37fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:39.541 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.559+0000 7fcfe37fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:39.541 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:24:39.543 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:39.548 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.564+0000 7fd00103b640 10 librados: watch_flush enter 2024-03-29T11:24:39.548 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.564+0000 7fd00103b640 10 librados: watch_flush exit 2024-03-29T11:24:39.548 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.564+0000 7fd00103b640 20 client.4397.objecter shutdown clearing up homeless session... 2024-03-29T11:24:39.548 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.564+0000 7fd00103b640 10 client.4397.objecter successfully canceled tick 2024-03-29T11:24:39.548 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.566+0000 7fd00103b640 1 -- 172.21.15.86:0/2639566565 >> v1:172.21.15.196:6816/588110318 conn(0x7fcfe806df30 legacy=0x7fcfe80703f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:39.549 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.566+0000 7fd00103b640 10 monclient: shutdown 2024-03-29T11:24:39.549 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.566+0000 7fd00103b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:39.549 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.566+0000 7fd00103b640 1 -- 172.21.15.86:0/2639566565 >> v1:172.21.15.86:6790/0 conn(0x7fcffc1457f0 legacy=0x7fcffc143e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:39.549 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.566+0000 7fd00103b640 1 -- 172.21.15.86:0/2639566565 shutdown_connections 2024-03-29T11:24:39.549 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.566+0000 7fd00103b640 1 -- 172.21.15.86:0/2639566565 >> 172.21.15.86:0/2639566565 conn(0x7fcffc066a00 msgr2=0x7fcffc13ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:39.550 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.568+0000 7fd00103b640 1 -- 172.21.15.86:0/2639566565 shutdown_connections 2024-03-29T11:24:39.550 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.568+0000 7fd00103b640 1 -- 172.21.15.86:0/2639566565 wait complete. 2024-03-29T11:24:39.550 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:39.568+0000 7fd00103b640 1 librados: shutdown 2024-03-29T11:24:39.568 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:46.374 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:46.527 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.545+0000 7f6d1e0b3640 10 monclient: get_monmap_and_config 2024-03-29T11:24:46.527 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.545+0000 7f6d1e0b3640 10 monclient: build_initial_monmap 2024-03-29T11:24:46.527 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.545+0000 7f6d1e0b3640 10 monclient: monmap: 2024-03-29T11:24:46.528 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:24:46.528 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:46.528 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:24:46.546241+0000 2024-03-29T11:24:46.528 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:24:46.546241+0000 2024-03-29T11:24:46.528 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:46.528 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:46.528 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:24:46.528 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:24:46.528 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:24:46.528 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:46.529 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.547+0000 7f6d1e0b3640 1 Processor -- start 2024-03-29T11:24:46.529 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.547+0000 7f6d1e0b3640 1 -- start start 2024-03-29T11:24:46.529 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.547+0000 7f6d1e0b3640 10 monclient: init 2024-03-29T11:24:46.530 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.548+0000 7f6d1e0b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:46.530 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.548+0000 7f6d1e0b3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:24:46.530 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.548+0000 7f6d1e0b3640 10 monclient(hunting): picked mon.noname-a con 0x7f6d180b4890 addr v1:172.21.15.86:6789/0 2024-03-29T11:24:46.530 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.548+0000 7f6d1e0b3640 10 monclient(hunting): picked mon.noname-b con 0x7f6d1813c040 addr v1:172.21.15.86:6790/0 2024-03-29T11:24:46.531 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.548+0000 7f6d1e0b3640 10 monclient(hunting): picked mon.noname-c con 0x7f6d181427e0 addr v1:172.21.15.196:6789/0 2024-03-29T11:24:46.531 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.548+0000 7f6d1e0b3640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d180b42f0 con 0x7f6d180b4890 2024-03-29T11:24:46.531 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.548+0000 7f6d1e0b3640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d180aa7a0 con 0x7f6d181427e0 2024-03-29T11:24:46.531 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.548+0000 7f6d1e0b3640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d180aaa10 con 0x7f6d1813c040 2024-03-29T11:24:46.531 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.548+0000 7f6d1e0b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:46.531 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.548+0000 7f6d1e0b3640 10 monclient(hunting): authenticate will time out at 1110.866896s 2024-03-29T11:24:46.535 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.552+0000 7f6d17fff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f6d181427e0 0x7f6d1814d880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:35603/0 (socket says 172.21.15.86:35603) 2024-03-29T11:24:46.538 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.552+0000 7f6d17fff640 1 -- 172.21.15.86:0/4111415293 learned_addr learned my addr 172.21.15.86:0/4111415293 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:24:46.538 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.553+0000 7f6d167fc640 1 -- 172.21.15.86:0/4111415293 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 904579379 0 0) 0x7f6d180aa7a0 con 0x7f6d181427e0 2024-03-29T11:24:46.538 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.553+0000 7f6d167fc640 10 monclient(hunting): _negotiate 2024-03-29T11:24:46.538 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.553+0000 7f6d167fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:46.538 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.553+0000 7f6d167fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:46.538 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.553+0000 7f6d167fc640 10 monclient(hunting): my global_id is 4345 2024-03-29T11:24:46.538 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.553+0000 7f6d167fc640 1 -- 172.21.15.86:0/4111415293 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d00003520 con 0x7f6d181427e0 2024-03-29T11:24:46.538 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.553+0000 7f6d167fc640 1 -- 172.21.15.86:0/4111415293 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 97598304 0 0) 0x7f6d00003520 con 0x7f6d181427e0 2024-03-29T11:24:46.539 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.553+0000 7f6d167fc640 1 -- 172.21.15.86:0/4111415293 >> v1:172.21.15.86:6790/0 conn(0x7f6d1813c040 legacy=0x7f6d181420d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:46.539 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.554+0000 7f6d167fc640 1 -- 172.21.15.86:0/4111415293 >> v1:172.21.15.86:6789/0 conn(0x7f6d180b4890 legacy=0x7f6d1813b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:24:46.539 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.554+0000 7f6d167fc640 10 monclient: _finish_hunting 0 2024-03-29T11:24:46.539 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.554+0000 7f6d167fc640 1 monclient: found mon.noname-c 2024-03-29T11:24:46.539 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.554+0000 7f6d167fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:46.539 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.554+0000 7f6d167fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T11:24:46.539 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.554+0000 7f6d167fc640 1 -- 172.21.15.86:0/4111415293 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d180aacf0 con 0x7f6d181427e0 2024-03-29T11:24:46.539 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.554+0000 7f6d167fc640 10 monclient: _finish_auth 0 2024-03-29T11:24:46.539 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.554+0000 7f6d167fc640 10 monclient: _check_auth_tickets 2024-03-29T11:24:46.539 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.554+0000 7f6d167fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:46.539 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.554+0000 7f6d167fc640 1 -- 172.21.15.86:0/4111415293 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f6d10001a10 con 0x7f6d181427e0 2024-03-29T11:24:46.539 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.554+0000 7f6d167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:46.539 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.554+0000 7f6d167fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:46.539 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.554+0000 7f6d167fc640 10 monclient: dump: 2024-03-29T11:24:46.539 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:24:46.540 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:46.540 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:46.540 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:46.540 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:46.540 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:46.540 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:24:46.540 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:24:46.540 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:24:46.540 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:46.541 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.554+0000 7f6d167fc640 1 -- 172.21.15.86:0/4111415293 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d10001c30 con 0x7f6d181427e0 2024-03-29T11:24:46.541 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.554+0000 7f6d167fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:46.541 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.554+0000 7f6d167fc640 1 -- 172.21.15.86:0/4111415293 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f6d10002000 con 0x7f6d181427e0 2024-03-29T11:24:46.541 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.554+0000 7f6d167fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:46.541 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.554+0000 7f6d167fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:46.541 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.555+0000 7f6d167fc640 10 monclient: dump: 2024-03-29T11:24:46.541 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:24:46.541 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:46.541 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:46.541 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:46.541 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:46.541 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:46.541 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:24:46.541 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:24:46.541 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:24:46.542 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:46.542 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.555+0000 7f6d1e0b3640 5 monclient: authenticate success, global_id 4345 2024-03-29T11:24:46.542 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.555+0000 7f6d1e0b3640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:46.542 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.555+0000 7f6d1e0b3640 4 set_mon_vals no callback set 2024-03-29T11:24:46.542 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.556+0000 7f6d1e0b3640 10 monclient: shutdown 2024-03-29T11:24:46.542 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.556+0000 7f6d1e0b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:46.542 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.556+0000 7f6d1e0b3640 1 -- 172.21.15.86:0/4111415293 >> v1:172.21.15.196:6789/0 conn(0x7f6d181427e0 legacy=0x7f6d1814d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:46.543 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.557+0000 7f6d1e0b3640 1 -- 172.21.15.86:0/4111415293 shutdown_connections 2024-03-29T11:24:46.543 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.557+0000 7f6d1e0b3640 1 -- 172.21.15.86:0/4111415293 >> 172.21.15.86:0/4111415293 conn(0x7f6d1806d180 msgr2=0x7f6d18066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:46.543 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.557+0000 7f6d1e0b3640 1 -- 172.21.15.86:0/4111415293 shutdown_connections 2024-03-29T11:24:46.543 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.557+0000 7f6d1e0b3640 1 -- 172.21.15.86:0/4111415293 wait complete. 2024-03-29T11:24:46.543 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.558+0000 7f6d1e0b3640 10 monclient: build_initial_monmap 2024-03-29T11:24:46.543 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.558+0000 7f6d1e0b3640 10 monclient: monmap: 2024-03-29T11:24:46.543 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:24:46.544 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:46.544 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:24:46.544 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:24:46.544 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:46.544 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:46.544 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:24:46.544 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:24:46.544 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:24:46.544 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:46.544 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.558+0000 7f6d1e0b3640 1 librados: starting msgr at 2024-03-29T11:24:46.544 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.558+0000 7f6d1e0b3640 1 librados: starting objecter 2024-03-29T11:24:46.544 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.558+0000 7f6d1e0b3640 1 Processor -- start 2024-03-29T11:24:46.544 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.558+0000 7f6d1e0b3640 1 -- start start 2024-03-29T11:24:46.544 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.558+0000 7f6d1e0b3640 1 librados: setting wanted keys 2024-03-29T11:24:46.544 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.558+0000 7f6d1e0b3640 1 librados: calling monclient init 2024-03-29T11:24:46.545 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.558+0000 7f6d1e0b3640 10 monclient: init 2024-03-29T11:24:46.545 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.559+0000 7f6d1e0b3640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:46.545 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.559+0000 7f6d1e0b3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:24:46.545 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.559+0000 7f6d1e0b3640 10 monclient(hunting): picked mon.noname-b con 0x7f6d180b4890 addr v1:172.21.15.196:6789/0 2024-03-29T11:24:46.548 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.562+0000 7f6d1e0b3640 10 monclient(hunting): picked mon.noname-a con 0x7f6d1813c040 addr v1:172.21.15.86:6789/0 2024-03-29T11:24:46.550 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.566+0000 7f6d177fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f6d180b4890 0x7f6d181e3e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:35689/0 (socket says 172.21.15.86:35689) 2024-03-29T11:24:46.551 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.566+0000 7f6d1e0b3640 10 monclient(hunting): picked mon.noname-c con 0x7f6d181427e0 addr v1:172.21.15.86:6790/0 2024-03-29T11:24:46.551 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.566+0000 7f6d1e0b3640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d18070870 con 0x7f6d1813c040 2024-03-29T11:24:46.551 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.566+0000 7f6d1e0b3640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d181e1300 con 0x7f6d180b4890 2024-03-29T11:24:46.551 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.566+0000 7f6d1e0b3640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d181e38f0 con 0x7f6d181427e0 2024-03-29T11:24:46.551 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.566+0000 7f6d1e0b3640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:46.551 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.566+0000 7f6d1e0b3640 10 monclient(hunting): authenticate will time out at 1110.884934s 2024-03-29T11:24:46.551 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.567+0000 7f6d177fe640 1 -- 172.21.15.86:0/2314102774 learned_addr learned my addr 172.21.15.86:0/2314102774 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:24:46.551 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.567+0000 7f6cfffff640 10 client.?.objecter ms_handle_connect 0x7f6d180b4890 2024-03-29T11:24:46.551 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.567+0000 7f6cfffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:46.551 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.567+0000 7f6cfffff640 1 -- 172.21.15.86:0/2314102774 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2728371407 0 0) 0x7f6d181e1300 con 0x7f6d180b4890 2024-03-29T11:24:46.551 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.567+0000 7f6cfffff640 10 monclient(hunting): _negotiate 2024-03-29T11:24:46.551 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.567+0000 7f6cfffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:46.551 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.567+0000 7f6cfffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:46.551 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.567+0000 7f6cfffff640 10 monclient(hunting): my global_id is 4348 2024-03-29T11:24:46.552 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.567+0000 7f6cfffff640 1 -- 172.21.15.86:0/2314102774 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6d04004eb0 con 0x7f6d180b4890 2024-03-29T11:24:46.552 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.568+0000 7f6cfffff640 1 -- 172.21.15.86:0/2314102774 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1993459771 0 0) 0x7f6d04004eb0 con 0x7f6d180b4890 2024-03-29T11:24:46.552 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.568+0000 7f6cfffff640 1 -- 172.21.15.86:0/2314102774 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6d181e1300 con 0x7f6d180b4890 2024-03-29T11:24:46.552 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.568+0000 7f6cfffff640 1 -- 172.21.15.86:0/2314102774 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f6d080017d0 con 0x7f6d180b4890 2024-03-29T11:24:46.552 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.568+0000 7f6cfffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:46.552 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.568+0000 7f6cfffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:46.552 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.568+0000 7f6cfffff640 10 monclient(hunting): dump: 2024-03-29T11:24:46.552 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:24:46.552 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:46.552 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:46.552 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:46.552 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:46.552 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:46.552 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:24:46.552 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:24:46.553 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:24:46.553 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:46.553 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.568+0000 7f6cfffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:46.557 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.572+0000 7f6cfffff640 10 client.?.objecter ms_handle_connect 0x7f6d1813c040 2024-03-29T11:24:46.557 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.572+0000 7f6cfffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:46.558 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.572+0000 7f6cfffff640 1 -- 172.21.15.86:0/2314102774 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3432523215 0 0) 0x7f6d181e1300 con 0x7f6d180b4890 2024-03-29T11:24:46.558 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.572+0000 7f6cfffff640 1 -- 172.21.15.86:0/2314102774 >> v1:172.21.15.86:6790/0 conn(0x7f6d181427e0 legacy=0x7f6d181e0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:46.558 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.572+0000 7f6cfffff640 1 -- 172.21.15.86:0/2314102774 >> v1:172.21.15.86:6789/0 conn(0x7f6d1813c040 legacy=0x7f6d181e04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:46.558 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.572+0000 7f6cfffff640 10 monclient: _finish_hunting 0 2024-03-29T11:24:46.558 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.572+0000 7f6cfffff640 1 monclient: found mon.b 2024-03-29T11:24:46.558 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.572+0000 7f6cfffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:46.558 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.572+0000 7f6cfffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:24:46.558 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.572+0000 7f6cfffff640 1 -- 172.21.15.86:0/2314102774 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d18138f90 con 0x7f6d180b4890 2024-03-29T11:24:46.558 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.572+0000 7f6cfffff640 10 monclient: _finish_auth 0 2024-03-29T11:24:46.558 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.572+0000 7f6cfffff640 10 monclient: _check_auth_tickets 2024-03-29T11:24:46.558 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.572+0000 7f6cfffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:46.558 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.572+0000 7f6d1e0b3640 5 monclient: authenticate success, global_id 4348 2024-03-29T11:24:46.558 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.572+0000 7f6d1e0b3640 10 monclient: _renew_subs 2024-03-29T11:24:46.559 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.572+0000 7f6d1e0b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:24:46.559 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.572+0000 7f6d1e0b3640 1 -- 172.21.15.86:0/2314102774 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d181e4520 con 0x7f6d180b4890 2024-03-29T11:24:46.559 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.572+0000 7f6d1e0b3640 10 client.4348.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:46.559 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.572+0000 7f6d1e0b3640 10 monclient: _renew_subs 2024-03-29T11:24:46.559 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.572+0000 7f6d1e0b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:24:46.559 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.572+0000 7f6d1e0b3640 1 -- 172.21.15.86:0/2314102774 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d181ef4a0 con 0x7f6d180b4890 2024-03-29T11:24:46.559 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.572+0000 7f6d1e0b3640 1 librados: init done 2024-03-29T11:24:46.559 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.573+0000 7f6cfffff640 1 -- 172.21.15.86:0/2314102774 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6d08001930 con 0x7f6d180b4890 2024-03-29T11:24:46.559 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.573+0000 7f6cfffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:46.559 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.573+0000 7f6cfffff640 1 -- 172.21.15.86:0/2314102774 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f6d08004a40 con 0x7f6d180b4890 2024-03-29T11:24:46.559 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.573+0000 7f6cfffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:46.559 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.573+0000 7f6cfffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:46.559 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.573+0000 7f6cfffff640 10 monclient: dump: 2024-03-29T11:24:46.559 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:24:46.559 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:46.559 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:46.560 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:46.560 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:46.560 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:46.560 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:24:46.560 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:24:46.560 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:24:46.560 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:46.560 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.575+0000 7f6cfffff640 1 -- 172.21.15.86:0/2314102774 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f6d0801aa10 con 0x7f6d180b4890 2024-03-29T11:24:46.560 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.575+0000 7f6d15ffb640 4 set_mon_vals no callback set 2024-03-29T11:24:46.560 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.575+0000 7f6d1e0b3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:46.561 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.576+0000 7f6d1e0b3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:46.561 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.576+0000 7f6d1e0b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:24:46.562 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.576+0000 7f6d1e0b3640 1 -- 172.21.15.86:0/2314102774 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d1804f730 con 0x7f6d180b4890 2024-03-29T11:24:46.562 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.577+0000 7f6cfffff640 1 -- 172.21.15.86:0/2314102774 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f6d08084a80 con 0x7f6d180b4890 2024-03-29T11:24:46.562 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.577+0000 7f6cfffff640 10 client.4348.objecter ms_dispatch 0x7f6d18000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:46.562 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.577+0000 7f6cfffff640 3 client.4348.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:46.562 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.577+0000 7f6cfffff640 3 client.4348.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:46.562 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.577+0000 7f6cfffff640 20 client.4348.objecter dump_active .. 0 homeless 2024-03-29T11:24:46.562 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.578+0000 7f6d16ffd640 10 monclient: get_auth_request con 0x7f6d0406dd60 auth_method 0 2024-03-29T11:24:46.562 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.578+0000 7f6cfffff640 10 client.4348.objecter ms_handle_connect 0x7f6d0406dd60 2024-03-29T11:24:46.567 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.584+0000 7f6cfffff640 1 -- 172.21.15.86:0/2314102774 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6d08050190 con 0x7f6d180b4890 2024-03-29T11:24:46.567 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.585+0000 7f6cfffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:46.567 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.585+0000 7f6cfffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:46.880 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.897+0000 7f6d1e0b3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:46.881 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.897+0000 7f6d1e0b3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:46.881 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.897+0000 7f6d1e0b3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:24:46.881 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.897+0000 7f6d1e0b3640 1 -- 172.21.15.86:0/2314102774 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6d18074d80 con 0x7f6d180b4890 2024-03-29T11:24:46.881 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.897+0000 7f6cfffff640 1 -- 172.21.15.86:0/2314102774 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f6d08052290 con 0x7f6d180b4890 2024-03-29T11:24:46.881 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.897+0000 7f6cfffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:46.881 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.897+0000 7f6cfffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:46.882 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:24:46.882 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:46.884 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.901+0000 7f6d1e0b3640 10 librados: watch_flush enter 2024-03-29T11:24:46.884 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.902+0000 7f6d1e0b3640 10 librados: watch_flush exit 2024-03-29T11:24:46.884 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.902+0000 7f6d1e0b3640 20 client.4348.objecter shutdown clearing up homeless session... 2024-03-29T11:24:46.884 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.902+0000 7f6d1e0b3640 10 client.4348.objecter successfully canceled tick 2024-03-29T11:24:46.885 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.902+0000 7f6d1e0b3640 1 -- 172.21.15.86:0/2314102774 >> v1:172.21.15.196:6816/588110318 conn(0x7f6d0406dd60 legacy=0x7f6d04070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:46.885 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.902+0000 7f6d1e0b3640 10 monclient: shutdown 2024-03-29T11:24:46.885 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.902+0000 7f6d1e0b3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:46.885 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.902+0000 7f6d1e0b3640 1 -- 172.21.15.86:0/2314102774 >> v1:172.21.15.196:6789/0 conn(0x7f6d180b4890 legacy=0x7f6d181e3e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:46.886 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.902+0000 7f6d1e0b3640 1 -- 172.21.15.86:0/2314102774 shutdown_connections 2024-03-29T11:24:46.886 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.902+0000 7f6d1e0b3640 1 -- 172.21.15.86:0/2314102774 >> 172.21.15.86:0/2314102774 conn(0x7f6d1806d180 msgr2=0x7f6d180b82d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:46.888 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.905+0000 7f6d1e0b3640 1 -- 172.21.15.86:0/2314102774 shutdown_connections 2024-03-29T11:24:46.889 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.906+0000 7f6d1e0b3640 1 -- 172.21.15.86:0/2314102774 wait complete. 2024-03-29T11:24:46.889 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:46.906+0000 7f6d1e0b3640 1 librados: shutdown 2024-03-29T11:24:46.914 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:24:53.721 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:24:53.870 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.887+0000 7f1bae3a8640 10 monclient: get_monmap_and_config 2024-03-29T11:24:53.870 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.887+0000 7f1bae3a8640 10 monclient: build_initial_monmap 2024-03-29T11:24:53.870 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.887+0000 7f1bae3a8640 10 monclient: monmap: 2024-03-29T11:24:53.870 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:24:53.870 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:53.870 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:24:53.888500+0000 2024-03-29T11:24:53.870 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:24:53.888500+0000 2024-03-29T11:24:53.871 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:53.871 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:53.871 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:24:53.871 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:24:53.871 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:24:53.871 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:53.874 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.889+0000 7f1bae3a8640 1 Processor -- start 2024-03-29T11:24:53.874 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.892+0000 7f1bae3a8640 1 -- start start 2024-03-29T11:24:53.875 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.892+0000 7f1bae3a8640 10 monclient: init 2024-03-29T11:24:53.876 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.893+0000 7f1bae3a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:53.876 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.893+0000 7f1bae3a8640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:24:53.876 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.893+0000 7f1bae3a8640 10 monclient(hunting): picked mon.noname-c con 0x7f1ba80b0770 addr v1:172.21.15.196:6789/0 2024-03-29T11:24:53.876 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.893+0000 7f1bae3a8640 10 monclient(hunting): picked mon.noname-b con 0x7f1ba814a700 addr v1:172.21.15.86:6790/0 2024-03-29T11:24:53.876 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.893+0000 7f1bae3a8640 10 monclient(hunting): picked mon.noname-a con 0x7f1ba80ac5a0 addr v1:172.21.15.86:6789/0 2024-03-29T11:24:53.876 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.893+0000 7f1bae3a8640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ba80aba70 con 0x7f1ba80ac5a0 2024-03-29T11:24:53.876 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.893+0000 7f1bae3a8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ba80aecb0 con 0x7f1ba80b0770 2024-03-29T11:24:53.876 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.893+0000 7f1bae3a8640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ba80af150 con 0x7f1ba814a700 2024-03-29T11:24:53.876 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.893+0000 7f1bae3a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:53.876 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.893+0000 7f1bae3a8640 10 monclient(hunting): authenticate will time out at 1118.211327s 2024-03-29T11:24:53.879 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.896+0000 7f1ba7fff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f1ba80b0770 0x7f1ba8149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:40416/0 (socket says 172.21.15.86:40416) 2024-03-29T11:24:53.879 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.896+0000 7f1ba7fff640 1 -- 172.21.15.86:0/185990507 learned_addr learned my addr 172.21.15.86:0/185990507 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:24:53.880 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.896+0000 7f1ba6ffd640 1 -- 172.21.15.86:0/185990507 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 94787823 0 0) 0x7f1ba80aecb0 con 0x7f1ba80b0770 2024-03-29T11:24:53.880 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.897+0000 7f1ba6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:53.880 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.897+0000 7f1ba6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:53.880 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.897+0000 7f1ba6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:53.880 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.897+0000 7f1ba6ffd640 10 monclient(hunting): my global_id is 4351 2024-03-29T11:24:53.880 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.897+0000 7f1ba6ffd640 1 -- 172.21.15.86:0/185990507 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b90003520 con 0x7f1ba80b0770 2024-03-29T11:24:53.880 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.897+0000 7f1ba6ffd640 1 -- 172.21.15.86:0/185990507 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1656511749 0 0) 0x7f1ba80af150 con 0x7f1ba814a700 2024-03-29T11:24:53.880 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.897+0000 7f1ba6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:24:53.880 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.897+0000 7f1ba6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:53.880 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.897+0000 7f1ba6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:53.880 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.897+0000 7f1ba6ffd640 10 monclient(hunting): my global_id is 4400 2024-03-29T11:24:53.880 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.897+0000 7f1ba6ffd640 1 -- 172.21.15.86:0/185990507 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ba80aecb0 con 0x7f1ba814a700 2024-03-29T11:24:53.880 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.897+0000 7f1ba6ffd640 1 -- 172.21.15.86:0/185990507 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3826941774 0 0) 0x7f1b90003520 con 0x7f1ba80b0770 2024-03-29T11:24:53.880 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.897+0000 7f1ba6ffd640 1 -- 172.21.15.86:0/185990507 >> v1:172.21.15.86:6790/0 conn(0x7f1ba814a700 legacy=0x7f1ba80abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:53.881 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.897+0000 7f1ba6ffd640 1 -- 172.21.15.86:0/185990507 >> v1:172.21.15.86:6789/0 conn(0x7f1ba80ac5a0 legacy=0x7f1ba8151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:53.881 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.897+0000 7f1ba6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:24:53.881 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.897+0000 7f1ba6ffd640 1 monclient: found mon.noname-c 2024-03-29T11:24:53.881 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.897+0000 7f1ba6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:53.881 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.897+0000 7f1ba6ffd640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T11:24:53.881 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.897+0000 7f1ba6ffd640 1 -- 172.21.15.86:0/185990507 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba80ac9d0 con 0x7f1ba80b0770 2024-03-29T11:24:53.881 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.897+0000 7f1ba6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:24:53.881 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.897+0000 7f1ba6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:24:53.881 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.897+0000 7f1ba6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:53.881 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.897+0000 7f1bae3a8640 5 monclient: authenticate success, global_id 4351 2024-03-29T11:24:53.881 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.897+0000 7f1bae3a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:53.881 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.897+0000 7f1ba6ffd640 1 -- 172.21.15.86:0/185990507 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f1b9c0016b0 con 0x7f1ba80b0770 2024-03-29T11:24:53.881 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.897+0000 7f1ba6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:53.881 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.897+0000 7f1ba6ffd640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:24:53.881 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.897+0000 7f1ba6ffd640 10 monclient: dump: 2024-03-29T11:24:53.882 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:24:53.882 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:53.882 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:53.882 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:53.882 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:53.882 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:53.882 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:24:53.882 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:24:53.882 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:24:53.882 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:53.882 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.897+0000 7f1bae3a8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:24:53.882 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.897+0000 7f1ba6ffd640 1 -- 172.21.15.86:0/185990507 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b9c001900 con 0x7f1ba80b0770 2024-03-29T11:24:53.882 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.897+0000 7f1ba6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:53.882 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.897+0000 7f1ba6ffd640 1 -- 172.21.15.86:0/185990507 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f1b9c001c90 con 0x7f1ba80b0770 2024-03-29T11:24:53.882 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.897+0000 7f1ba6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:53.883 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.897+0000 7f1ba6ffd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:53.883 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.897+0000 7f1ba6ffd640 10 monclient: dump: 2024-03-29T11:24:53.883 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:24:53.883 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:53.883 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:53.883 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:53.883 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:53.883 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:53.883 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:24:53.883 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:24:53.883 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:24:53.883 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:53.885 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.898+0000 7f1bae3a8640 10 monclient: get_monmap_and_config success 2024-03-29T11:24:53.885 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.898+0000 7f1bae3a8640 4 set_mon_vals no callback set 2024-03-29T11:24:53.885 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.899+0000 7f1bae3a8640 10 monclient: shutdown 2024-03-29T11:24:53.885 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.899+0000 7f1bae3a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:53.885 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.899+0000 7f1bae3a8640 1 -- 172.21.15.86:0/185990507 >> v1:172.21.15.196:6789/0 conn(0x7f1ba80b0770 legacy=0x7f1ba8149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:53.885 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.901+0000 7f1bae3a8640 1 -- 172.21.15.86:0/185990507 shutdown_connections 2024-03-29T11:24:53.885 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.901+0000 7f1bae3a8640 1 -- 172.21.15.86:0/185990507 >> 172.21.15.86:0/185990507 conn(0x7f1ba806b660 msgr2=0x7f1ba806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:53.885 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.901+0000 7f1bae3a8640 1 -- 172.21.15.86:0/185990507 shutdown_connections 2024-03-29T11:24:53.885 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.901+0000 7f1bae3a8640 1 -- 172.21.15.86:0/185990507 wait complete. 2024-03-29T11:24:53.885 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.902+0000 7f1bae3a8640 10 monclient: build_initial_monmap 2024-03-29T11:24:53.885 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.902+0000 7f1bae3a8640 10 monclient: monmap: 2024-03-29T11:24:53.885 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:24:53.885 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:24:53.886 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:24:53.886 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:24:53.886 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:24:53.886 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:53.886 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:24:53.886 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:24:53.886 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:24:53.886 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:53.886 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.902+0000 7f1bae3a8640 1 librados: starting msgr at 2024-03-29T11:24:53.886 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.902+0000 7f1bae3a8640 1 librados: starting objecter 2024-03-29T11:24:53.886 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.902+0000 7f1bae3a8640 1 Processor -- start 2024-03-29T11:24:53.890 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.903+0000 7f1bae3a8640 1 -- start start 2024-03-29T11:24:53.890 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.903+0000 7f1bae3a8640 1 librados: setting wanted keys 2024-03-29T11:24:53.890 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.903+0000 7f1bae3a8640 1 librados: calling monclient init 2024-03-29T11:24:53.890 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.903+0000 7f1bae3a8640 10 monclient: init 2024-03-29T11:24:53.890 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.903+0000 7f1bae3a8640 10 monclient: _reopen_session rank -1 2024-03-29T11:24:53.890 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.903+0000 7f1bae3a8640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:24:53.890 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.903+0000 7f1bae3a8640 10 monclient(hunting): picked mon.noname-c con 0x7f1ba80ac5a0 addr v1:172.21.15.86:6790/0 2024-03-29T11:24:53.890 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.903+0000 7f1bae3a8640 10 monclient(hunting): picked mon.noname-a con 0x7f1ba80b0770 addr v1:172.21.15.86:6789/0 2024-03-29T11:24:53.890 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.904+0000 7f1bae3a8640 10 monclient(hunting): picked mon.noname-b con 0x7f1ba814a700 addr v1:172.21.15.196:6789/0 2024-03-29T11:24:53.890 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.904+0000 7f1bae3a8640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ba8144d90 con 0x7f1ba80b0770 2024-03-29T11:24:53.890 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.904+0000 7f1bac945640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f1ba814a700 0x7f1ba81e6e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:45272/0 (socket says 172.21.15.86:45272) 2024-03-29T11:24:53.891 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.904+0000 7f1bac945640 1 -- 172.21.15.86:0/2981733964 learned_addr learned my addr 172.21.15.86:0/2981733964 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:24:53.891 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.904+0000 7f1bae3a8640 1 -- 172.21.15.86:0/2981733964 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ba81264e0 con 0x7f1ba814a700 2024-03-29T11:24:53.891 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.904+0000 7f1bae3a8640 1 -- 172.21.15.86:0/2981733964 --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ba8126670 con 0x7f1ba80ac5a0 2024-03-29T11:24:53.891 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.904+0000 7f1bae3a8640 10 monclient(hunting): _renew_subs 2024-03-29T11:24:53.891 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.904+0000 7f1bae3a8640 10 monclient(hunting): authenticate will time out at 1118.222787s 2024-03-29T11:24:53.891 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.906+0000 7f1ba4ff9640 10 client.?.objecter ms_handle_connect 0x7f1ba814a700 2024-03-29T11:24:53.891 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.906+0000 7f1ba4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:53.891 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.906+0000 7f1ba4ff9640 1 -- 172.21.15.86:0/2981733964 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4078663864 0 0) 0x7f1ba81264e0 con 0x7f1ba814a700 2024-03-29T11:24:53.891 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.906+0000 7f1ba4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:24:53.891 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.906+0000 7f1ba4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:24:53.891 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.906+0000 7f1ba4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:24:53.891 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.906+0000 7f1ba4ff9640 10 monclient(hunting): my global_id is 4354 2024-03-29T11:24:53.891 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.906+0000 7f1ba4ff9640 1 -- 172.21.15.86:0/2981733964 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b94004eb0 con 0x7f1ba814a700 2024-03-29T11:24:53.891 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.906+0000 7f1ba4ff9640 10 client.?.objecter ms_handle_connect 0x7f1ba80ac5a0 2024-03-29T11:24:53.891 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.906+0000 7f1ba4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:24:53.892 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.906+0000 7f1ba4ff9640 1 -- 172.21.15.86:0/2981733964 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1024352161 0 0) 0x7f1b94004eb0 con 0x7f1ba814a700 2024-03-29T11:24:53.892 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.906+0000 7f1ba4ff9640 1 -- 172.21.15.86:0/2981733964 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ba81264e0 con 0x7f1ba814a700 2024-03-29T11:24:53.892 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.906+0000 7f1ba4ff9640 1 -- 172.21.15.86:0/2981733964 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f1b980019b0 con 0x7f1ba814a700 2024-03-29T11:24:53.892 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.906+0000 7f1ba4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:53.892 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.906+0000 7f1ba4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:24:53.892 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.906+0000 7f1ba4ff9640 10 monclient(hunting): dump: 2024-03-29T11:24:53.892 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:24:53.892 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:53.892 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:53.892 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:53.892 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:53.892 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:53.892 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:24:53.892 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:24:53.892 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:24:53.893 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:53.893 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.906+0000 7f1ba4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:24:53.893 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.907+0000 7f1ba4ff9640 1 -- 172.21.15.86:0/2981733964 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 491083905 0 0) 0x7f1ba81264e0 con 0x7f1ba814a700 2024-03-29T11:24:53.893 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.907+0000 7f1ba4ff9640 1 -- 172.21.15.86:0/2981733964 >> v1:172.21.15.86:6790/0 conn(0x7f1ba80ac5a0 legacy=0x7f1ba81deae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:53.893 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.907+0000 7f1ba4ff9640 1 -- 172.21.15.86:0/2981733964 >> v1:172.21.15.86:6789/0 conn(0x7f1ba80b0770 legacy=0x7f1ba8144010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:24:53.896 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.909+0000 7f1ba4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:24:53.896 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.909+0000 7f1ba4ff9640 1 monclient: found mon.b 2024-03-29T11:24:53.897 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.909+0000 7f1ba4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:24:53.897 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.909+0000 7f1ba4ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:24:53.897 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.909+0000 7f1ba4ff9640 1 -- 172.21.15.86:0/2981733964 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba8125aa0 con 0x7f1ba814a700 2024-03-29T11:24:53.897 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.909+0000 7f1ba4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:24:53.897 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.909+0000 7f1ba4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:24:53.897 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.909+0000 7f1ba4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:24:53.897 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.909+0000 7f1bae3a8640 5 monclient: authenticate success, global_id 4354 2024-03-29T11:24:53.897 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.909+0000 7f1bae3a8640 10 monclient: _renew_subs 2024-03-29T11:24:53.897 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.909+0000 7f1bae3a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:24:53.897 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.909+0000 7f1bae3a8640 1 -- 172.21.15.86:0/2981733964 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ba8125d50 con 0x7f1ba814a700 2024-03-29T11:24:53.897 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.909+0000 7f1bae3a8640 10 client.4354.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:24:53.897 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.909+0000 7f1bae3a8640 10 monclient: _renew_subs 2024-03-29T11:24:53.897 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.909+0000 7f1bae3a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:24:53.897 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.909+0000 7f1bae3a8640 1 -- 172.21.15.86:0/2981733964 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ba81df3d0 con 0x7f1ba814a700 2024-03-29T11:24:53.897 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.909+0000 7f1bae3a8640 1 librados: init done 2024-03-29T11:24:53.898 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.910+0000 7f1ba4ff9640 1 -- 172.21.15.86:0/2981733964 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1b98001c70 con 0x7f1ba814a700 2024-03-29T11:24:53.898 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.910+0000 7f1ba4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:24:53.898 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.910+0000 7f1ba67fc640 4 set_mon_vals no callback set 2024-03-29T11:24:53.898 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.911+0000 7f1ba4ff9640 1 -- 172.21.15.86:0/2981733964 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f1b98003a60 con 0x7f1ba814a700 2024-03-29T11:24:53.898 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.911+0000 7f1ba4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:24:53.898 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.911+0000 7f1ba4ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:24:53.898 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.911+0000 7f1ba4ff9640 10 monclient: dump: 2024-03-29T11:24:53.898 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:24:53.898 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:24:53.898 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:53.898 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:24:53.898 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:24:53.898 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:24:53.898 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:24:53.899 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:24:53.899 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:24:53.899 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:24:53.899 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.912+0000 7f1ba4ff9640 1 -- 172.21.15.86:0/2981733964 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f1b98003c40 con 0x7f1ba814a700 2024-03-29T11:24:53.899 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.912+0000 7f1ba4ff9640 1 -- 172.21.15.86:0/2981733964 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f1b98084ae0 con 0x7f1ba814a700 2024-03-29T11:24:53.899 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.913+0000 7f1ba4ff9640 10 client.4354.objecter ms_dispatch 0x7f1ba8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:24:53.899 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.913+0000 7f1ba4ff9640 3 client.4354.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:24:53.899 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.913+0000 7f1ba4ff9640 3 client.4354.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:24:53.899 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.913+0000 7f1ba4ff9640 20 client.4354.objecter dump_active .. 0 homeless 2024-03-29T11:24:53.899 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.913+0000 7f1ba7fff640 10 monclient: get_auth_request con 0x7f1b9406dc40 auth_method 0 2024-03-29T11:24:53.899 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.913+0000 7f1bae3a8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:53.899 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.913+0000 7f1bae3a8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:53.899 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.913+0000 7f1bae3a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:24:53.900 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.913+0000 7f1bae3a8640 1 -- 172.21.15.86:0/2981733964 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b88005080 con 0x7f1ba814a700 2024-03-29T11:24:53.900 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.914+0000 7f1ba4ff9640 10 client.4354.objecter ms_handle_connect 0x7f1b9406dc40 2024-03-29T11:24:53.906 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.923+0000 7f1ba4ff9640 1 -- 172.21.15.86:0/2981733964 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1b980503c0 con 0x7f1ba814a700 2024-03-29T11:24:53.906 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.923+0000 7f1ba4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:24:53.906 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:53.923+0000 7f1ba4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:24:54.240 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:54.257+0000 7f1bae3a8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:54.240 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:54.257+0000 7f1bae3a8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:54.240 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:54.257+0000 7f1bae3a8640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:24:54.240 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:54.257+0000 7f1bae3a8640 1 -- 172.21.15.86:0/2981733964 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b88005330 con 0x7f1ba814a700 2024-03-29T11:24:54.243 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:54.260+0000 7f1ba4ff9640 1 -- 172.21.15.86:0/2981733964 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f1b980522f0 con 0x7f1ba814a700 2024-03-29T11:24:54.243 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:54.260+0000 7f1ba4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:24:54.243 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:54.260+0000 7f1ba4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:24:54.243 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:24:54.243 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:24:54.246 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:54.263+0000 7f1bae3a8640 10 librados: watch_flush enter 2024-03-29T11:24:54.246 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:54.263+0000 7f1bae3a8640 10 librados: watch_flush exit 2024-03-29T11:24:54.246 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:54.263+0000 7f1bae3a8640 20 client.4354.objecter shutdown clearing up homeless session... 2024-03-29T11:24:54.246 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:54.263+0000 7f1bae3a8640 10 client.4354.objecter successfully canceled tick 2024-03-29T11:24:54.246 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:54.263+0000 7f1bae3a8640 1 -- 172.21.15.86:0/2981733964 >> v1:172.21.15.196:6816/588110318 conn(0x7f1b9406dc40 legacy=0x7f1b94070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:54.246 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:54.264+0000 7f1bae3a8640 10 monclient: shutdown 2024-03-29T11:24:54.246 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:54.264+0000 7f1bae3a8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:24:54.246 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:54.264+0000 7f1bae3a8640 1 -- 172.21.15.86:0/2981733964 >> v1:172.21.15.196:6789/0 conn(0x7f1ba814a700 legacy=0x7f1ba81e6e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:24:54.248 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:54.265+0000 7f1bae3a8640 1 -- 172.21.15.86:0/2981733964 shutdown_connections 2024-03-29T11:24:54.248 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:54.265+0000 7f1bae3a8640 1 -- 172.21.15.86:0/2981733964 >> 172.21.15.86:0/2981733964 conn(0x7f1ba806b660 msgr2=0x7f1ba8149510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:24:54.249 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:54.266+0000 7f1bae3a8640 1 -- 172.21.15.86:0/2981733964 shutdown_connections 2024-03-29T11:24:54.249 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:54.266+0000 7f1bae3a8640 1 -- 172.21.15.86:0/2981733964 wait complete. 2024-03-29T11:24:54.249 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:24:54.266+0000 7f1bae3a8640 1 librados: shutdown 2024-03-29T11:24:54.267 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:01.074 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:01.233 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.250+0000 7f450b315640 10 monclient: get_monmap_and_config 2024-03-29T11:25:01.234 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.250+0000 7f450b315640 10 monclient: build_initial_monmap 2024-03-29T11:25:01.234 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.250+0000 7f450b315640 10 monclient: monmap: 2024-03-29T11:25:01.234 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:25:01.234 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:01.234 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:25:01.251863+0000 2024-03-29T11:25:01.234 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:25:01.251863+0000 2024-03-29T11:25:01.234 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:01.234 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:01.234 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:25:01.234 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:25:01.235 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:25:01.235 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:01.235 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.251+0000 7f450b315640 1 Processor -- start 2024-03-29T11:25:01.235 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.252+0000 7f450b315640 1 -- start start 2024-03-29T11:25:01.235 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.252+0000 7f450b315640 10 monclient: init 2024-03-29T11:25:01.236 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.253+0000 7f450b315640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:01.236 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.253+0000 7f450b315640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:01.236 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.253+0000 7f450b315640 10 monclient(hunting): picked mon.noname-b con 0x7f45041457f0 addr v1:172.21.15.86:6790/0 2024-03-29T11:25:01.236 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.253+0000 7f450b315640 10 monclient(hunting): picked mon.noname-c con 0x7f450413f650 addr v1:172.21.15.196:6789/0 2024-03-29T11:25:01.236 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.253+0000 7f450b315640 10 monclient(hunting): picked mon.noname-a con 0x7f45040a9d70 addr v1:172.21.15.86:6789/0 2024-03-29T11:25:01.236 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.253+0000 7f450b315640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45040aaef0 con 0x7f45040a9d70 2024-03-29T11:25:01.236 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.253+0000 7f450b315640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4504145bd0 con 0x7f450413f650 2024-03-29T11:25:01.236 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.253+0000 7f450b315640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4504145eb0 con 0x7f45041457f0 2024-03-29T11:25:01.236 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.253+0000 7f450b315640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:01.236 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.253+0000 7f450b315640 10 monclient(hunting): authenticate will time out at 1125.571328s 2024-03-29T11:25:01.237 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.253+0000 7f45090b1640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f45041457f0 0x7f450413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:57360/0 (socket says 172.21.15.86:57360) 2024-03-29T11:25:01.237 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.253+0000 7f45090b1640 1 -- 172.21.15.86:0/293115670 learned_addr learned my addr 172.21.15.86:0/293115670 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:25:01.239 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.254+0000 7f4503fff640 1 -- 172.21.15.86:0/293115670 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 102753506 0 0) 0x7f4504145bd0 con 0x7f450413f650 2024-03-29T11:25:01.239 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.254+0000 7f4503fff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:01.239 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.254+0000 7f4503fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:01.239 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.254+0000 7f4503fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:01.239 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.254+0000 7f4503fff640 10 monclient(hunting): my global_id is 4357 2024-03-29T11:25:01.239 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.254+0000 7f4503fff640 1 -- 172.21.15.86:0/293115670 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44ec003520 con 0x7f450413f650 2024-03-29T11:25:01.240 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.254+0000 7f4503fff640 1 -- 172.21.15.86:0/293115670 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3716512421 0 0) 0x7f4504145eb0 con 0x7f45041457f0 2024-03-29T11:25:01.240 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.254+0000 7f4503fff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:01.240 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.254+0000 7f4503fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:01.240 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.254+0000 7f4503fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:01.240 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.254+0000 7f4503fff640 10 monclient(hunting): my global_id is 4406 2024-03-29T11:25:01.240 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.254+0000 7f4503fff640 1 -- 172.21.15.86:0/293115670 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4504145bd0 con 0x7f45041457f0 2024-03-29T11:25:01.240 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.254+0000 7f4503fff640 1 -- 172.21.15.86:0/293115670 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1142124653 0 0) 0x7f44ec003520 con 0x7f450413f650 2024-03-29T11:25:01.240 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.254+0000 7f4503fff640 1 -- 172.21.15.86:0/293115670 >> v1:172.21.15.86:6790/0 conn(0x7f45041457f0 legacy=0x7f450413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:01.240 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.254+0000 7f4503fff640 1 -- 172.21.15.86:0/293115670 >> v1:172.21.15.86:6789/0 conn(0x7f45040a9d70 legacy=0x7f450414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:01.240 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.254+0000 7f4503fff640 10 monclient: _finish_hunting 0 2024-03-29T11:25:01.240 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.254+0000 7f4503fff640 1 monclient: found mon.noname-c 2024-03-29T11:25:01.240 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.254+0000 7f4503fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:01.240 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.254+0000 7f4503fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T11:25:01.240 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.254+0000 7f4503fff640 1 -- 172.21.15.86:0/293115670 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4504146340 con 0x7f450413f650 2024-03-29T11:25:01.241 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.254+0000 7f4503fff640 10 monclient: _finish_auth 0 2024-03-29T11:25:01.241 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.254+0000 7f4503fff640 10 monclient: _check_auth_tickets 2024-03-29T11:25:01.241 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.254+0000 7f4503fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:01.241 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.254+0000 7f4503fff640 1 -- 172.21.15.86:0/293115670 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f44f8001690 con 0x7f450413f650 2024-03-29T11:25:01.241 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.254+0000 7f4503fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:01.241 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.254+0000 7f4503fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:01.241 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.254+0000 7f4503fff640 10 monclient: dump: 2024-03-29T11:25:01.241 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:01.241 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:01.241 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:01.241 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:01.241 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:01.241 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:01.242 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:01.242 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:01.242 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:01.242 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:01.242 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.254+0000 7f450b315640 5 monclient: authenticate success, global_id 4357 2024-03-29T11:25:01.242 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.254+0000 7f450b315640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:01.242 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.255+0000 7f4503fff640 1 -- 172.21.15.86:0/293115670 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44f80017f0 con 0x7f450413f650 2024-03-29T11:25:01.242 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.255+0000 7f4503fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:01.242 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.255+0000 7f4503fff640 1 -- 172.21.15.86:0/293115670 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f44f8001c00 con 0x7f450413f650 2024-03-29T11:25:01.242 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.255+0000 7f4503fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:01.242 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.255+0000 7f4503fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:01.242 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.255+0000 7f4503fff640 10 monclient: dump: 2024-03-29T11:25:01.243 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:01.243 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:01.243 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:01.243 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:01.243 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:01.243 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:01.243 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:01.243 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:01.243 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:01.243 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:01.243 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.255+0000 7f450b315640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:01.243 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.255+0000 7f450b315640 4 set_mon_vals no callback set 2024-03-29T11:25:01.243 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.256+0000 7f450b315640 10 monclient: shutdown 2024-03-29T11:25:01.243 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.256+0000 7f450b315640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:01.243 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.256+0000 7f450b315640 1 -- 172.21.15.86:0/293115670 >> v1:172.21.15.196:6789/0 conn(0x7f450413f650 legacy=0x7f45040b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:01.244 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.256+0000 7f450b315640 1 -- 172.21.15.86:0/293115670 shutdown_connections 2024-03-29T11:25:01.244 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.256+0000 7f450b315640 1 -- 172.21.15.86:0/293115670 >> 172.21.15.86:0/293115670 conn(0x7f4504066a00 msgr2=0x7f4504066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:01.244 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.256+0000 7f450b315640 1 -- 172.21.15.86:0/293115670 shutdown_connections 2024-03-29T11:25:01.244 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.256+0000 7f450b315640 1 -- 172.21.15.86:0/293115670 wait complete. 2024-03-29T11:25:01.248 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.257+0000 7f450b315640 10 monclient: build_initial_monmap 2024-03-29T11:25:01.248 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.257+0000 7f450b315640 10 monclient: monmap: 2024-03-29T11:25:01.248 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:25:01.248 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:01.248 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:25:01.248 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:25:01.249 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:01.249 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:01.249 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:25:01.249 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:25:01.249 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:25:01.249 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:01.249 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.257+0000 7f450b315640 1 librados: starting msgr at 2024-03-29T11:25:01.250 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.257+0000 7f450b315640 1 librados: starting objecter 2024-03-29T11:25:01.250 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.257+0000 7f450b315640 1 Processor -- start 2024-03-29T11:25:01.250 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.257+0000 7f450b315640 1 -- start start 2024-03-29T11:25:01.250 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.257+0000 7f450b315640 1 librados: setting wanted keys 2024-03-29T11:25:01.250 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.257+0000 7f450b315640 1 librados: calling monclient init 2024-03-29T11:25:01.250 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.257+0000 7f450b315640 10 monclient: init 2024-03-29T11:25:01.250 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.258+0000 7f450b315640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:01.251 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.258+0000 7f450b315640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:01.251 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.258+0000 7f450b315640 10 monclient(hunting): picked mon.noname-a con 0x7f45040a9d70 addr v1:172.21.15.86:6789/0 2024-03-29T11:25:01.251 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.258+0000 7f450b315640 10 monclient(hunting): picked mon.noname-b con 0x7f450413f650 addr v1:172.21.15.196:6789/0 2024-03-29T11:25:01.251 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.258+0000 7f450b315640 10 monclient(hunting): picked mon.noname-c con 0x7f45041457f0 addr v1:172.21.15.86:6790/0 2024-03-29T11:25:01.251 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.258+0000 7f450b315640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4504132740 con 0x7f45040a9d70 2024-03-29T11:25:01.251 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.258+0000 7f450b315640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4504071c70 con 0x7f450413f650 2024-03-29T11:25:01.251 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.258+0000 7f450b315640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45040ab850 con 0x7f45041457f0 2024-03-29T11:25:01.251 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.259+0000 7f450b315640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:01.252 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.259+0000 7f450b315640 10 monclient(hunting): authenticate will time out at 1125.577014s 2024-03-29T11:25:01.252 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.259+0000 7f45088b0640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f450413f650 0x7f4504131920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:55510/0 (socket says 172.21.15.86:55510) 2024-03-29T11:25:01.252 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.259+0000 7f45088b0640 1 -- 172.21.15.86:0/3109328369 learned_addr learned my addr 172.21.15.86:0/3109328369 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:25:01.252 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.259+0000 7f4501ffb640 10 client.?.objecter ms_handle_connect 0x7f450413f650 2024-03-29T11:25:01.252 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.259+0000 7f4501ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:01.252 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.259+0000 7f4501ffb640 1 -- 172.21.15.86:0/3109328369 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 526080882 0 0) 0x7f4504071c70 con 0x7f450413f650 2024-03-29T11:25:01.252 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.259+0000 7f4501ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:01.252 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.259+0000 7f4501ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:01.252 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.259+0000 7f4501ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:01.252 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.259+0000 7f4501ffb640 10 monclient(hunting): my global_id is 4360 2024-03-29T11:25:01.252 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.259+0000 7f4501ffb640 1 -- 172.21.15.86:0/3109328369 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44f0004eb0 con 0x7f450413f650 2024-03-29T11:25:01.252 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.259+0000 7f4501ffb640 10 client.?.objecter ms_handle_connect 0x7f45041457f0 2024-03-29T11:25:01.253 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.259+0000 7f4501ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:01.253 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.259+0000 7f4501ffb640 10 client.?.objecter ms_handle_connect 0x7f45040a9d70 2024-03-29T11:25:01.253 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.259+0000 7f4501ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:01.253 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.260+0000 7f4501ffb640 1 -- 172.21.15.86:0/3109328369 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 302994640 0 0) 0x7f44f0004eb0 con 0x7f450413f650 2024-03-29T11:25:01.253 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.260+0000 7f4501ffb640 1 -- 172.21.15.86:0/3109328369 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f4504071c70 con 0x7f450413f650 2024-03-29T11:25:01.253 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.260+0000 7f4501ffb640 1 -- 172.21.15.86:0/3109328369 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f44f8001690 con 0x7f450413f650 2024-03-29T11:25:01.253 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.260+0000 7f4501ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:01.253 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.260+0000 7f4501ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:01.253 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.260+0000 7f4501ffb640 10 monclient(hunting): dump: 2024-03-29T11:25:01.253 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:01.253 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:01.253 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:01.253 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:01.253 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:01.253 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:01.254 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:01.254 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:01.254 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:01.254 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:01.254 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.260+0000 7f4501ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:01.254 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.260+0000 7f4501ffb640 1 -- 172.21.15.86:0/3109328369 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2957925031 0 0) 0x7f4504132740 con 0x7f45040a9d70 2024-03-29T11:25:01.254 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.260+0000 7f4501ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:01.254 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.260+0000 7f4501ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:01.254 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.260+0000 7f4501ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:01.254 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.260+0000 7f4501ffb640 10 monclient(hunting): my global_id is 4389 2024-03-29T11:25:01.254 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.260+0000 7f4501ffb640 1 -- 172.21.15.86:0/3109328369 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44f0004eb0 con 0x7f45040a9d70 2024-03-29T11:25:01.254 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.260+0000 7f4501ffb640 1 -- 172.21.15.86:0/3109328369 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2422996805 0 0) 0x7f45040ab850 con 0x7f45041457f0 2024-03-29T11:25:01.254 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.260+0000 7f4501ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:01.255 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.260+0000 7f4501ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:01.255 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.260+0000 7f4501ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:01.255 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.260+0000 7f4501ffb640 10 monclient(hunting): my global_id is 4409 2024-03-29T11:25:01.255 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.260+0000 7f4501ffb640 1 -- 172.21.15.86:0/3109328369 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4504132740 con 0x7f45041457f0 2024-03-29T11:25:01.255 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.260+0000 7f4501ffb640 1 -- 172.21.15.86:0/3109328369 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2490951529 0 0) 0x7f44f0004eb0 con 0x7f45040a9d70 2024-03-29T11:25:01.255 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.260+0000 7f4501ffb640 1 -- 172.21.15.86:0/3109328369 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f45040ab850 con 0x7f45040a9d70 2024-03-29T11:25:01.255 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.260+0000 7f4501ffb640 1 -- 172.21.15.86:0/3109328369 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f44fc001a30 con 0x7f45040a9d70 2024-03-29T11:25:01.255 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.260+0000 7f4501ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:01.255 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.260+0000 7f4501ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:01.255 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.260+0000 7f4501ffb640 10 monclient(hunting): dump: 2024-03-29T11:25:01.255 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:01.255 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:01.255 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:01.255 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:01.256 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:01.256 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:01.256 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:01.256 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:01.256 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:01.256 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:01.256 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.261+0000 7f4501ffb640 1 -- 172.21.15.86:0/3109328369 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3384769691 0 0) 0x7f4504071c70 con 0x7f450413f650 2024-03-29T11:25:01.256 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.261+0000 7f4501ffb640 1 -- 172.21.15.86:0/3109328369 >> v1:172.21.15.86:6790/0 conn(0x7f45041457f0 legacy=0x7f4504132030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:01.256 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.261+0000 7f4501ffb640 1 -- 172.21.15.86:0/3109328369 >> v1:172.21.15.86:6789/0 conn(0x7f45040a9d70 legacy=0x7f45041dc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:01.256 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.261+0000 7f4501ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:25:01.256 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.261+0000 7f4501ffb640 1 monclient: found mon.b 2024-03-29T11:25:01.256 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.261+0000 7f4501ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:01.256 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.261+0000 7f4501ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:25:01.256 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.261+0000 7f4501ffb640 1 -- 172.21.15.86:0/3109328369 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f450413c890 con 0x7f450413f650 2024-03-29T11:25:01.257 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.261+0000 7f4501ffb640 10 monclient: _finish_auth 0 2024-03-29T11:25:01.257 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.261+0000 7f4501ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:25:01.257 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.261+0000 7f4501ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:01.257 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.261+0000 7f450b315640 5 monclient: authenticate success, global_id 4360 2024-03-29T11:25:01.257 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.261+0000 7f450b315640 10 monclient: _renew_subs 2024-03-29T11:25:01.257 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.261+0000 7f450b315640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:25:01.257 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.261+0000 7f450b315640 1 -- 172.21.15.86:0/3109328369 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4504070a90 con 0x7f450413f650 2024-03-29T11:25:01.257 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.261+0000 7f450b315640 10 client.4360.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:01.257 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.261+0000 7f450b315640 10 monclient: _renew_subs 2024-03-29T11:25:01.257 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.261+0000 7f450b315640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:25:01.257 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.261+0000 7f450b315640 1 -- 172.21.15.86:0/3109328369 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f45040706e0 con 0x7f450413f650 2024-03-29T11:25:01.257 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.261+0000 7f450b315640 1 librados: init done 2024-03-29T11:25:01.257 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.261+0000 7f4501ffb640 1 -- 172.21.15.86:0/3109328369 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f44f80038a0 con 0x7f450413f650 2024-03-29T11:25:01.257 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.261+0000 7f4501ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:01.257 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.261+0000 7f4501ffb640 1 -- 172.21.15.86:0/3109328369 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f44f8003c70 con 0x7f450413f650 2024-03-29T11:25:01.258 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.261+0000 7f4501ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:01.258 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.261+0000 7f4501ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:01.258 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.261+0000 7f4501ffb640 10 monclient: dump: 2024-03-29T11:25:01.258 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:01.258 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:01.258 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:01.258 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:01.258 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:01.258 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:01.258 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:01.258 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:01.258 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:01.258 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:01.258 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.261+0000 7f45037fe640 4 set_mon_vals no callback set 2024-03-29T11:25:01.259 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.263+0000 7f450b315640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:01.259 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.263+0000 7f450b315640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:01.259 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.263+0000 7f450b315640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:25:01.259 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.263+0000 7f450b315640 1 -- 172.21.15.86:0/3109328369 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4504075f10 con 0x7f450413f650 2024-03-29T11:25:01.259 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.264+0000 7f4501ffb640 1 -- 172.21.15.86:0/3109328369 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f44f8003dd0 con 0x7f450413f650 2024-03-29T11:25:01.259 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.264+0000 7f4501ffb640 1 -- 172.21.15.86:0/3109328369 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f44f8084c10 con 0x7f450413f650 2024-03-29T11:25:01.259 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.264+0000 7f4501ffb640 10 client.4360.objecter ms_dispatch 0x7f4504000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:01.259 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.264+0000 7f4501ffb640 3 client.4360.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:01.259 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.264+0000 7f4501ffb640 3 client.4360.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:01.259 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.264+0000 7f45090b1640 10 monclient: get_auth_request con 0x7f44f006e050 auth_method 0 2024-03-29T11:25:01.259 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.264+0000 7f4501ffb640 20 client.4360.objecter dump_active .. 0 homeless 2024-03-29T11:25:01.259 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.265+0000 7f4501ffb640 10 client.4360.objecter ms_handle_connect 0x7f44f006e050 2024-03-29T11:25:01.260 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.271+0000 7f4501ffb640 1 -- 172.21.15.86:0/3109328369 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f44f80504f0 con 0x7f450413f650 2024-03-29T11:25:01.260 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.271+0000 7f4501ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:01.260 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.271+0000 7f4501ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:01.650 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.667+0000 7f450b315640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:01.650 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.667+0000 7f450b315640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:01.650 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.667+0000 7f450b315640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:25:01.650 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.667+0000 7f450b315640 1 -- 172.21.15.86:0/3109328369 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4504070c20 con 0x7f450413f650 2024-03-29T11:25:01.651 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.668+0000 7f4501ffb640 1 -- 172.21.15.86:0/3109328369 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f44f80504f0 con 0x7f450413f650 2024-03-29T11:25:01.651 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.668+0000 7f4501ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:01.651 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.668+0000 7f4501ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:01.651 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:25:01.651 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:01.655 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.672+0000 7f450b315640 10 librados: watch_flush enter 2024-03-29T11:25:01.655 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.672+0000 7f450b315640 10 librados: watch_flush exit 2024-03-29T11:25:01.655 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.672+0000 7f450b315640 20 client.4360.objecter shutdown clearing up homeless session... 2024-03-29T11:25:01.655 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.672+0000 7f450b315640 10 client.4360.objecter successfully canceled tick 2024-03-29T11:25:01.655 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.672+0000 7f450b315640 1 -- 172.21.15.86:0/3109328369 >> v1:172.21.15.196:6816/588110318 conn(0x7f44f006e050 legacy=0x7f44f0070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:01.656 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.672+0000 7f450b315640 10 monclient: shutdown 2024-03-29T11:25:01.656 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.672+0000 7f450b315640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:01.656 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.672+0000 7f450b315640 1 -- 172.21.15.86:0/3109328369 >> v1:172.21.15.196:6789/0 conn(0x7f450413f650 legacy=0x7f4504131920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:01.656 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.672+0000 7f450b315640 1 -- 172.21.15.86:0/3109328369 shutdown_connections 2024-03-29T11:25:01.656 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.673+0000 7f450b315640 1 -- 172.21.15.86:0/3109328369 >> 172.21.15.86:0/3109328369 conn(0x7f4504066a00 msgr2=0x7f450413d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:01.656 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.673+0000 7f450b315640 1 -- 172.21.15.86:0/3109328369 shutdown_connections 2024-03-29T11:25:01.657 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.673+0000 7f450b315640 1 -- 172.21.15.86:0/3109328369 wait complete. 2024-03-29T11:25:01.657 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:01.673+0000 7f450b315640 1 librados: shutdown 2024-03-29T11:25:01.672 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:08.480 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:08.628 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.645+0000 7f79634e6640 10 monclient: get_monmap_and_config 2024-03-29T11:25:08.628 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.645+0000 7f79634e6640 10 monclient: build_initial_monmap 2024-03-29T11:25:08.628 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.645+0000 7f79634e6640 10 monclient: monmap: 2024-03-29T11:25:08.628 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:25:08.629 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:08.629 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:25:08.646127+0000 2024-03-29T11:25:08.629 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:25:08.646127+0000 2024-03-29T11:25:08.629 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:08.629 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:08.629 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:25:08.629 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:25:08.629 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:25:08.629 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:08.630 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.646+0000 7f79634e6640 1 Processor -- start 2024-03-29T11:25:08.630 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.646+0000 7f79634e6640 1 -- start start 2024-03-29T11:25:08.630 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.646+0000 7f79634e6640 10 monclient: init 2024-03-29T11:25:08.632 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.647+0000 7f79634e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:08.632 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.647+0000 7f79634e6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:08.632 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.647+0000 7f79634e6640 10 monclient(hunting): picked mon.noname-c con 0x7f795c070170 addr v1:172.21.15.196:6789/0 2024-03-29T11:25:08.632 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.647+0000 7f79634e6640 10 monclient(hunting): picked mon.noname-b con 0x7f795c0713f0 addr v1:172.21.15.86:6790/0 2024-03-29T11:25:08.632 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.647+0000 7f79634e6640 10 monclient(hunting): picked mon.noname-a con 0x7f795c070550 addr v1:172.21.15.86:6789/0 2024-03-29T11:25:08.632 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.647+0000 7f79634e6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f795c13dbd0 con 0x7f795c070550 2024-03-29T11:25:08.632 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.647+0000 7f79634e6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f795c13ff70 con 0x7f795c070170 2024-03-29T11:25:08.632 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.647+0000 7f79634e6640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f795c140250 con 0x7f795c0713f0 2024-03-29T11:25:08.632 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.647+0000 7f79634e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:08.632 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.647+0000 7f79634e6640 10 monclient(hunting): authenticate will time out at 1132.965501s 2024-03-29T11:25:08.632 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.647+0000 7f7961a83640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f795c070550 0x7f795c14b7d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:38024/0 (socket says 172.21.15.86:38024) 2024-03-29T11:25:08.632 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.647+0000 7f7961a83640 1 -- 172.21.15.86:0/1894252326 learned_addr learned my addr 172.21.15.86:0/1894252326 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:25:08.632 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.648+0000 7f7953fff640 1 -- 172.21.15.86:0/1894252326 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1823916024 0 0) 0x7f7954001490 con 0x7f795c070550 2024-03-29T11:25:08.633 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.648+0000 7f7953fff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:08.633 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.648+0000 7f7953fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:08.633 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.648+0000 7f7953fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:08.633 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.648+0000 7f7953fff640 10 monclient(hunting): my global_id is 4392 2024-03-29T11:25:08.633 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.648+0000 7f7953fff640 1 -- 172.21.15.86:0/1894252326 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7948003520 con 0x7f795c070550 2024-03-29T11:25:08.633 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.648+0000 7f7953fff640 1 -- 172.21.15.86:0/1894252326 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3350418600 0 0) 0x7f795c13ff70 con 0x7f795c070170 2024-03-29T11:25:08.633 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.648+0000 7f7953fff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:08.633 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.648+0000 7f7953fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:08.633 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.648+0000 7f7953fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:08.633 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.648+0000 7f7953fff640 10 monclient(hunting): my global_id is 4363 2024-03-29T11:25:08.633 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.648+0000 7f7953fff640 1 -- 172.21.15.86:0/1894252326 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7954001490 con 0x7f795c070170 2024-03-29T11:25:08.633 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.648+0000 7f7953fff640 1 -- 172.21.15.86:0/1894252326 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3345024552 0 0) 0x7f7948003520 con 0x7f795c070550 2024-03-29T11:25:08.633 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.648+0000 7f7953fff640 1 -- 172.21.15.86:0/1894252326 >> v1:172.21.15.86:6790/0 conn(0x7f795c0713f0 legacy=0x7f795c1406e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:08.635 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.651+0000 7f7953fff640 1 -- 172.21.15.86:0/1894252326 >> v1:172.21.15.196:6789/0 conn(0x7f795c070170 legacy=0x7f795c13d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:08.635 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.651+0000 7f7953fff640 10 monclient: _finish_hunting 0 2024-03-29T11:25:08.635 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.651+0000 7f7953fff640 1 monclient: found mon.noname-a 2024-03-29T11:25:08.635 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.651+0000 7f7953fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:08.636 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.651+0000 7f7953fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.86:6789/0 2024-03-29T11:25:08.636 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.651+0000 7f7953fff640 1 -- 172.21.15.86:0/1894252326 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f795c140df0 con 0x7f795c070550 2024-03-29T11:25:08.636 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.651+0000 7f7953fff640 10 monclient: _finish_auth 0 2024-03-29T11:25:08.636 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.651+0000 7f7953fff640 10 monclient: _check_auth_tickets 2024-03-29T11:25:08.636 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.651+0000 7f7953fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:08.636 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.651+0000 7f7953fff640 1 -- 172.21.15.86:0/1894252326 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f7954001980 con 0x7f795c070550 2024-03-29T11:25:08.636 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.651+0000 7f7953fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:08.636 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.651+0000 7f7953fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:08.636 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.651+0000 7f7953fff640 10 monclient: dump: 2024-03-29T11:25:08.636 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:08.636 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:08.636 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:08.636 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:08.636 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:08.636 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:08.637 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:08.637 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:08.637 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:08.637 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:08.637 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.651+0000 7f7953fff640 1 -- 172.21.15.86:0/1894252326 <== mon.0 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7954001ae0 con 0x7f795c070550 2024-03-29T11:25:08.637 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.651+0000 7f7953fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:08.637 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.651+0000 7f7953fff640 1 -- 172.21.15.86:0/1894252326 <== mon.0 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f7954001ef0 con 0x7f795c070550 2024-03-29T11:25:08.637 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.652+0000 7f7953fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:08.637 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.652+0000 7f7953fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:08.637 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.652+0000 7f7953fff640 10 monclient: dump: 2024-03-29T11:25:08.637 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:08.637 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:08.637 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:08.637 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:08.638 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:08.638 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:08.638 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:08.638 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:08.638 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:08.638 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:08.638 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.652+0000 7f79634e6640 5 monclient: authenticate success, global_id 4392 2024-03-29T11:25:08.638 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.652+0000 7f79634e6640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:08.638 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.652+0000 7f79634e6640 4 set_mon_vals no callback set 2024-03-29T11:25:08.640 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.653+0000 7f79634e6640 10 monclient: shutdown 2024-03-29T11:25:08.640 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.653+0000 7f79634e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:08.640 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.653+0000 7f79634e6640 1 -- 172.21.15.86:0/1894252326 >> v1:172.21.15.86:6789/0 conn(0x7f795c070550 legacy=0x7f795c14b7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:08.640 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.653+0000 7f79634e6640 1 -- 172.21.15.86:0/1894252326 shutdown_connections 2024-03-29T11:25:08.640 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.653+0000 7f79634e6640 1 -- 172.21.15.86:0/1894252326 >> 172.21.15.86:0/1894252326 conn(0x7f795c067610 msgr2=0x7f795c0b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:08.640 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.654+0000 7f79634e6640 1 -- 172.21.15.86:0/1894252326 shutdown_connections 2024-03-29T11:25:08.640 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.654+0000 7f79634e6640 1 -- 172.21.15.86:0/1894252326 wait complete. 2024-03-29T11:25:08.640 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.655+0000 7f79634e6640 10 monclient: build_initial_monmap 2024-03-29T11:25:08.640 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.655+0000 7f79634e6640 10 monclient: monmap: 2024-03-29T11:25:08.641 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:25:08.641 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:08.641 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:25:08.641 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:25:08.641 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:08.641 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:08.641 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:25:08.641 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:25:08.641 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:25:08.641 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:08.641 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.656+0000 7f79634e6640 1 librados: starting msgr at 2024-03-29T11:25:08.641 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.656+0000 7f79634e6640 1 librados: starting objecter 2024-03-29T11:25:08.641 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.656+0000 7f79634e6640 1 Processor -- start 2024-03-29T11:25:08.641 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.656+0000 7f79634e6640 1 -- start start 2024-03-29T11:25:08.642 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.656+0000 7f79634e6640 1 librados: setting wanted keys 2024-03-29T11:25:08.642 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.656+0000 7f79634e6640 1 librados: calling monclient init 2024-03-29T11:25:08.642 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.656+0000 7f79634e6640 10 monclient: init 2024-03-29T11:25:08.649 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.657+0000 7f79634e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:08.649 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.657+0000 7f79634e6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:08.649 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.657+0000 7f79634e6640 10 monclient(hunting): picked mon.noname-c con 0x7f795c070170 addr v1:172.21.15.86:6790/0 2024-03-29T11:25:08.649 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.657+0000 7f79634e6640 10 monclient(hunting): picked mon.noname-b con 0x7f795c070550 addr v1:172.21.15.196:6789/0 2024-03-29T11:25:08.649 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.657+0000 7f79634e6640 10 monclient(hunting): picked mon.noname-a con 0x7f795c0713f0 addr v1:172.21.15.86:6789/0 2024-03-29T11:25:08.649 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.657+0000 7f7960a81640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f795c070550 0x7f795c133b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:50484/0 (socket says 172.21.15.86:50484) 2024-03-29T11:25:08.649 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.657+0000 7f7960a81640 1 -- 172.21.15.86:0/1801964593 learned_addr learned my addr 172.21.15.86:0/1801964593 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:25:08.649 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.658+0000 7f79634e6640 1 -- 172.21.15.86:0/1801964593 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f795c1349b0 con 0x7f795c0713f0 2024-03-29T11:25:08.649 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.658+0000 7f79634e6640 1 -- 172.21.15.86:0/1801964593 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f795c13a890 con 0x7f795c070550 2024-03-29T11:25:08.649 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.658+0000 7f79634e6640 1 -- 172.21.15.86:0/1801964593 --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f795c13adb0 con 0x7f795c070170 2024-03-29T11:25:08.650 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.658+0000 7f79634e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:08.650 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.658+0000 7f79634e6640 10 monclient(hunting): authenticate will time out at 1132.976208s 2024-03-29T11:25:08.650 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.658+0000 7f7951ffb640 10 client.?.objecter ms_handle_connect 0x7f795c070550 2024-03-29T11:25:08.650 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.658+0000 7f7951ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:08.650 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.658+0000 7f7951ffb640 1 -- 172.21.15.86:0/1801964593 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3112961044 0 0) 0x7f795c13a890 con 0x7f795c070550 2024-03-29T11:25:08.650 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.658+0000 7f7951ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:08.650 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.658+0000 7f7951ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:08.650 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.658+0000 7f7951ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:08.650 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.658+0000 7f7951ffb640 10 monclient(hunting): my global_id is 4366 2024-03-29T11:25:08.650 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.658+0000 7f7951ffb640 1 -- 172.21.15.86:0/1801964593 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7940004eb0 con 0x7f795c070550 2024-03-29T11:25:08.650 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.658+0000 7f7951ffb640 10 client.?.objecter ms_handle_connect 0x7f795c0713f0 2024-03-29T11:25:08.650 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.658+0000 7f7951ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:08.650 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.658+0000 7f7951ffb640 1 -- 172.21.15.86:0/1801964593 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1091695291 0 0) 0x7f795c1349b0 con 0x7f795c0713f0 2024-03-29T11:25:08.651 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.658+0000 7f7951ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:08.651 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.658+0000 7f7951ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:08.651 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.658+0000 7f7951ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:08.651 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.658+0000 7f7951ffb640 10 monclient(hunting): my global_id is 4395 2024-03-29T11:25:08.651 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.658+0000 7f7951ffb640 1 -- 172.21.15.86:0/1801964593 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f795c13a890 con 0x7f795c0713f0 2024-03-29T11:25:08.651 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.660+0000 7f7951ffb640 1 -- 172.21.15.86:0/1801964593 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3778915501 0 0) 0x7f795c13a890 con 0x7f795c0713f0 2024-03-29T11:25:08.651 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.660+0000 7f7951ffb640 1 -- 172.21.15.86:0/1801964593 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f795c1349b0 con 0x7f795c0713f0 2024-03-29T11:25:08.651 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.660+0000 7f7951ffb640 1 -- 172.21.15.86:0/1801964593 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3430403805 0 0) 0x7f7940004eb0 con 0x7f795c070550 2024-03-29T11:25:08.651 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.660+0000 7f7951ffb640 1 -- 172.21.15.86:0/1801964593 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f795c13a890 con 0x7f795c070550 2024-03-29T11:25:08.651 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.660+0000 7f7951ffb640 1 -- 172.21.15.86:0/1801964593 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f7954001df0 con 0x7f795c0713f0 2024-03-29T11:25:08.651 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.660+0000 7f7951ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:08.651 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.660+0000 7f7951ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:08.651 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.660+0000 7f7951ffb640 10 monclient(hunting): dump: 2024-03-29T11:25:08.651 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:08.652 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:08.652 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:08.652 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:08.652 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:08.652 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:08.652 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:08.652 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:08.652 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:08.652 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:08.652 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.660+0000 7f7951ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:08.652 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.660+0000 7f7951ffb640 1 -- 172.21.15.86:0/1801964593 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f79580018b0 con 0x7f795c070550 2024-03-29T11:25:08.652 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.660+0000 7f7951ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:08.652 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.660+0000 7f7951ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:08.652 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.660+0000 7f7951ffb640 10 monclient(hunting): dump: 2024-03-29T11:25:08.652 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:08.653 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:08.653 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:08.653 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:08.653 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:08.653 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:08.653 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:08.653 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:08.653 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:08.653 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:08.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.660+0000 7f7951ffb640 1 -- 172.21.15.86:0/1801964593 <== mon.0 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1422972975 0 0) 0x7f795c1349b0 con 0x7f795c0713f0 2024-03-29T11:25:08.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.660+0000 7f7951ffb640 1 -- 172.21.15.86:0/1801964593 >> v1:172.21.15.86:6790/0 conn(0x7f795c070170 legacy=0x7f795c1d67c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:08.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.661+0000 7f7951ffb640 1 -- 172.21.15.86:0/1801964593 >> v1:172.21.15.196:6789/0 conn(0x7f795c070550 legacy=0x7f795c133b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:08.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.661+0000 7f7951ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:25:08.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.661+0000 7f7951ffb640 1 monclient: found mon.a 2024-03-29T11:25:08.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.661+0000 7f7951ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:08.654 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.661+0000 7f7951ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:25:08.654 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.661+0000 7f7951ffb640 1 -- 172.21.15.86:0/1801964593 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f795c139c40 con 0x7f795c0713f0 2024-03-29T11:25:08.654 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.661+0000 7f7951ffb640 10 monclient: _finish_auth 0 2024-03-29T11:25:08.654 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.661+0000 7f7951ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:25:08.654 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.661+0000 7f7951ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:08.654 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.661+0000 7f79634e6640 5 monclient: authenticate success, global_id 4395 2024-03-29T11:25:08.654 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.661+0000 7f79634e6640 10 monclient: _renew_subs 2024-03-29T11:25:08.654 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.661+0000 7f79634e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:25:08.654 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.661+0000 7f79634e6640 1 -- 172.21.15.86:0/1801964593 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f795c13a4b0 con 0x7f795c0713f0 2024-03-29T11:25:08.654 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.662+0000 7f79634e6640 10 client.4395.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:08.654 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.662+0000 7f79634e6640 10 monclient: _renew_subs 2024-03-29T11:25:08.654 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.662+0000 7f79634e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:25:08.654 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.662+0000 7f79634e6640 1 -- 172.21.15.86:0/1801964593 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f795c1d70b0 con 0x7f795c0713f0 2024-03-29T11:25:08.655 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.662+0000 7f79634e6640 1 librados: init done 2024-03-29T11:25:08.655 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.662+0000 7f7951ffb640 1 -- 172.21.15.86:0/1801964593 <== mon.0 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f7954001f50 con 0x7f795c0713f0 2024-03-29T11:25:08.655 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.662+0000 7f7951ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:08.655 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.662+0000 7f7951ffb640 1 -- 172.21.15.86:0/1801964593 <== mon.0 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f7954004fb0 con 0x7f795c0713f0 2024-03-29T11:25:08.655 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.662+0000 7f7951ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:08.655 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.662+0000 7f7951ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:08.655 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.662+0000 7f7951ffb640 10 monclient: dump: 2024-03-29T11:25:08.655 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:08.655 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:08.655 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:08.655 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:08.655 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:08.655 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:08.655 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:08.655 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:08.656 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:08.656 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:08.656 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.662+0000 7f79537fe640 4 set_mon_vals no callback set 2024-03-29T11:25:08.656 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.664+0000 7f79634e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:08.656 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.664+0000 7f79634e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:08.656 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.664+0000 7f79634e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:25:08.656 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.664+0000 7f79634e6640 1 -- 172.21.15.86:0/1801964593 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f793c005080 con 0x7f795c0713f0 2024-03-29T11:25:08.656 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.665+0000 7f7951ffb640 1 -- 172.21.15.86:0/1801964593 <== mon.0 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f7954005110 con 0x7f795c0713f0 2024-03-29T11:25:08.656 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.665+0000 7f7951ffb640 1 -- 172.21.15.86:0/1801964593 <== mon.0 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f7954084df0 con 0x7f795c0713f0 2024-03-29T11:25:08.656 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.665+0000 7f7951ffb640 10 client.4395.objecter ms_dispatch 0x7f795c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:08.656 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.665+0000 7f7951ffb640 3 client.4395.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:08.656 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.665+0000 7f7951ffb640 3 client.4395.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:08.656 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.665+0000 7f7951ffb640 20 client.4395.objecter dump_active .. 0 homeless 2024-03-29T11:25:08.656 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.665+0000 7f7961282640 10 monclient: get_auth_request con 0x7f794006de00 auth_method 0 2024-03-29T11:25:08.656 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.666+0000 7f7951ffb640 10 client.4395.objecter ms_handle_connect 0x7f794006de00 2024-03-29T11:25:08.658 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.675+0000 7f7951ffb640 1 -- 172.21.15.86:0/1801964593 <== mon.0 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f79540506d0 con 0x7f795c0713f0 2024-03-29T11:25:08.658 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.675+0000 7f7951ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:08.658 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.675+0000 7f7951ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:08.967 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.984+0000 7f79634e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:08.967 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.984+0000 7f79634e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:08.967 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.984+0000 7f79634e6640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:25:08.967 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.984+0000 7f79634e6640 1 -- 172.21.15.86:0/1801964593 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f793c005330 con 0x7f795c0713f0 2024-03-29T11:25:08.968 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.984+0000 7f7951ffb640 1 -- 172.21.15.86:0/1801964593 <== mon.0 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f7954052600 con 0x7f795c0713f0 2024-03-29T11:25:08.968 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.985+0000 7f7951ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:08.968 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.985+0000 7f7951ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:08.968 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:25:08.968 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:08.971 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.987+0000 7f79634e6640 10 librados: watch_flush enter 2024-03-29T11:25:08.972 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.987+0000 7f79634e6640 10 librados: watch_flush exit 2024-03-29T11:25:08.972 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.987+0000 7f79634e6640 20 client.4395.objecter shutdown clearing up homeless session... 2024-03-29T11:25:08.972 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.987+0000 7f79634e6640 10 client.4395.objecter successfully canceled tick 2024-03-29T11:25:08.972 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.988+0000 7f79634e6640 1 -- 172.21.15.86:0/1801964593 >> v1:172.21.15.196:6816/588110318 conn(0x7f794006de00 legacy=0x7f79400702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:08.972 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.988+0000 7f79634e6640 10 monclient: shutdown 2024-03-29T11:25:08.972 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.988+0000 7f79634e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:08.972 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.988+0000 7f79634e6640 1 -- 172.21.15.86:0/1801964593 >> v1:172.21.15.86:6789/0 conn(0x7f795c0713f0 legacy=0x7f795c1342a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:08.972 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.988+0000 7f79634e6640 1 -- 172.21.15.86:0/1801964593 shutdown_connections 2024-03-29T11:25:08.972 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.988+0000 7f79634e6640 1 -- 172.21.15.86:0/1801964593 >> 172.21.15.86:0/1801964593 conn(0x7f795c067610 msgr2=0x7f795c13d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:08.972 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.988+0000 7f79634e6640 1 -- 172.21.15.86:0/1801964593 shutdown_connections 2024-03-29T11:25:08.972 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.988+0000 7f79634e6640 1 -- 172.21.15.86:0/1801964593 wait complete. 2024-03-29T11:25:08.972 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:08.989+0000 7f79634e6640 1 librados: shutdown 2024-03-29T11:25:08.985 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:15.793 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:15.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.944+0000 7f54b8a45640 10 monclient: get_monmap_and_config 2024-03-29T11:25:15.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.944+0000 7f54b8a45640 10 monclient: build_initial_monmap 2024-03-29T11:25:15.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.944+0000 7f54b8a45640 10 monclient: monmap: 2024-03-29T11:25:15.928 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:25:15.928 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:15.928 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:25:15.945726+0000 2024-03-29T11:25:15.928 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:25:15.945726+0000 2024-03-29T11:25:15.928 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:15.929 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:15.929 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:25:15.929 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:25:15.929 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:25:15.929 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:15.929 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.945+0000 7f54b8a45640 1 Processor -- start 2024-03-29T11:25:15.929 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.945+0000 7f54b8a45640 1 -- start start 2024-03-29T11:25:15.929 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.945+0000 7f54b8a45640 10 monclient: init 2024-03-29T11:25:15.930 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.946+0000 7f54b8a45640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:15.930 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.946+0000 7f54b8a45640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:15.930 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.946+0000 7f54b8a45640 10 monclient(hunting): picked mon.noname-c con 0x7f54b41422b0 addr v1:172.21.15.196:6789/0 2024-03-29T11:25:15.930 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.946+0000 7f54b8a45640 10 monclient(hunting): picked mon.noname-b con 0x7f54b4142690 addr v1:172.21.15.86:6790/0 2024-03-29T11:25:15.940 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.949+0000 7f54b259c640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f54b41422b0 0x7f54b413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:37390/0 (socket says 172.21.15.86:37390) 2024-03-29T11:25:15.941 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.951+0000 7f54b8a45640 10 monclient(hunting): picked mon.noname-a con 0x7f54b40b17d0 addr v1:172.21.15.86:6789/0 2024-03-29T11:25:15.941 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.951+0000 7f54b8a45640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54b4142c80 con 0x7f54b40b17d0 2024-03-29T11:25:15.941 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.951+0000 7f54b8a45640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54b40aae40 con 0x7f54b41422b0 2024-03-29T11:25:15.941 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.951+0000 7f54b8a45640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54b413fbd0 con 0x7f54b4142690 2024-03-29T11:25:15.941 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.951+0000 7f54b8a45640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:15.941 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.951+0000 7f54b8a45640 10 monclient(hunting): authenticate will time out at 1140.269260s 2024-03-29T11:25:15.941 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.951+0000 7f54b259c640 1 -- 172.21.15.86:0/2546905731 learned_addr learned my addr 172.21.15.86:0/2546905731 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:25:15.941 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.952+0000 7f54b159a640 1 -- 172.21.15.86:0/2546905731 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4119944681 0 0) 0x7f54b413fbd0 con 0x7f54b4142690 2024-03-29T11:25:15.941 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.952+0000 7f54b159a640 10 monclient(hunting): _negotiate 2024-03-29T11:25:15.941 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.952+0000 7f54b159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:15.942 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.952+0000 7f54b159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:15.942 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.952+0000 7f54b159a640 10 monclient(hunting): my global_id is 4415 2024-03-29T11:25:15.942 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.952+0000 7f54b159a640 1 -- 172.21.15.86:0/2546905731 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f549c003520 con 0x7f54b4142690 2024-03-29T11:25:15.942 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.952+0000 7f54b159a640 1 -- 172.21.15.86:0/2546905731 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2673746970 0 0) 0x7f54b40aae40 con 0x7f54b41422b0 2024-03-29T11:25:15.942 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.952+0000 7f54b159a640 10 monclient(hunting): _negotiate 2024-03-29T11:25:15.942 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.952+0000 7f54b159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:15.942 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.952+0000 7f54b159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:15.942 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.952+0000 7f54b159a640 10 monclient(hunting): my global_id is 4369 2024-03-29T11:25:15.942 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.952+0000 7f54b159a640 1 -- 172.21.15.86:0/2546905731 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54b413fbd0 con 0x7f54b41422b0 2024-03-29T11:25:15.943 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.952+0000 7f54b159a640 1 -- 172.21.15.86:0/2546905731 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3388764060 0 0) 0x7f549c003520 con 0x7f54b4142690 2024-03-29T11:25:15.943 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.952+0000 7f54b159a640 1 -- 172.21.15.86:0/2546905731 >> v1:172.21.15.196:6789/0 conn(0x7f54b41422b0 legacy=0x7f54b413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:15.943 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.952+0000 7f54b159a640 1 -- 172.21.15.86:0/2546905731 >> v1:172.21.15.86:6789/0 conn(0x7f54b40b17d0 legacy=0x7f54b40b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:15.943 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.953+0000 7f54b159a640 10 monclient: _finish_hunting 0 2024-03-29T11:25:15.943 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.953+0000 7f54b159a640 1 monclient: found mon.noname-b 2024-03-29T11:25:15.943 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.953+0000 7f54b159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:15.943 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.953+0000 7f54b159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:25:15.943 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.953+0000 7f54b159a640 1 -- 172.21.15.86:0/2546905731 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b4151940 con 0x7f54b4142690 2024-03-29T11:25:15.943 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.953+0000 7f54b159a640 10 monclient: _finish_auth 0 2024-03-29T11:25:15.943 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.954+0000 7f54b159a640 10 monclient: _check_auth_tickets 2024-03-29T11:25:15.944 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.954+0000 7f54b159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:15.944 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.954+0000 7f54b159a640 1 -- 172.21.15.86:0/2546905731 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f54a4001610 con 0x7f54b4142690 2024-03-29T11:25:15.944 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.954+0000 7f54b8a45640 5 monclient: authenticate success, global_id 4415 2024-03-29T11:25:15.944 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.954+0000 7f54b8a45640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:15.944 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.954+0000 7f54b159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:15.944 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.954+0000 7f54b159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:15.944 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.954+0000 7f54b159a640 10 monclient: dump: 2024-03-29T11:25:15.944 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:15.944 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:15.944 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:15.945 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:15.945 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:15.945 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:15.945 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:15.945 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:15.945 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:15.945 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:15.945 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.954+0000 7f54b159a640 1 -- 172.21.15.86:0/2546905731 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54a4001830 con 0x7f54b4142690 2024-03-29T11:25:15.945 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.954+0000 7f54b8a45640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:15.945 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.954+0000 7f54b159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:15.945 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.954+0000 7f54b159a640 1 -- 172.21.15.86:0/2546905731 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f54a4001bc0 con 0x7f54b4142690 2024-03-29T11:25:15.945 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.954+0000 7f54b8a45640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:15.946 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.954+0000 7f54b8a45640 4 set_mon_vals no callback set 2024-03-29T11:25:15.946 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.954+0000 7f54b159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:15.946 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.954+0000 7f54b159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:15.946 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.954+0000 7f54b159a640 10 monclient: dump: 2024-03-29T11:25:15.946 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:15.946 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:15.946 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:15.946 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:15.946 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:15.946 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:15.946 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:15.946 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:15.946 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:15.947 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:15.947 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.955+0000 7f54b8a45640 10 monclient: shutdown 2024-03-29T11:25:15.947 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.955+0000 7f54b8a45640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:15.947 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.955+0000 7f54b8a45640 1 -- 172.21.15.86:0/2546905731 >> v1:172.21.15.86:6790/0 conn(0x7f54b4142690 legacy=0x7f54b40b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:15.947 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.956+0000 7f54b8a45640 1 -- 172.21.15.86:0/2546905731 shutdown_connections 2024-03-29T11:25:15.947 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.956+0000 7f54b8a45640 1 -- 172.21.15.86:0/2546905731 >> 172.21.15.86:0/2546905731 conn(0x7f54b4066a00 msgr2=0x7f54b4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:15.951 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.958+0000 7f54b8a45640 1 -- 172.21.15.86:0/2546905731 shutdown_connections 2024-03-29T11:25:15.951 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.958+0000 7f54b8a45640 1 -- 172.21.15.86:0/2546905731 wait complete. 2024-03-29T11:25:15.951 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.959+0000 7f54b8a45640 10 monclient: build_initial_monmap 2024-03-29T11:25:15.951 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.959+0000 7f54b8a45640 10 monclient: monmap: 2024-03-29T11:25:15.951 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:25:15.951 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:15.951 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:25:15.951 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:25:15.952 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:15.952 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:15.952 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:25:15.952 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:25:15.952 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:25:15.952 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:15.952 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.959+0000 7f54b8a45640 1 librados: starting msgr at 2024-03-29T11:25:15.952 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.959+0000 7f54b8a45640 1 librados: starting objecter 2024-03-29T11:25:15.952 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.959+0000 7f54b8a45640 1 Processor -- start 2024-03-29T11:25:15.952 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.959+0000 7f54b8a45640 1 -- start start 2024-03-29T11:25:15.952 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.959+0000 7f54b8a45640 1 librados: setting wanted keys 2024-03-29T11:25:15.952 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.959+0000 7f54b8a45640 1 librados: calling monclient init 2024-03-29T11:25:15.952 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.959+0000 7f54b8a45640 10 monclient: init 2024-03-29T11:25:15.953 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.960+0000 7f54b8a45640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:15.953 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.960+0000 7f54b8a45640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:15.953 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.960+0000 7f54b8a45640 10 monclient(hunting): picked mon.noname-a con 0x7f54b40b17d0 addr v1:172.21.15.86:6789/0 2024-03-29T11:25:15.953 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.960+0000 7f54b8a45640 10 monclient(hunting): picked mon.noname-c con 0x7f54b41422b0 addr v1:172.21.15.86:6790/0 2024-03-29T11:25:15.953 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.961+0000 7f54b8a45640 10 monclient(hunting): picked mon.noname-b con 0x7f54b4142690 addr v1:172.21.15.196:6789/0 2024-03-29T11:25:15.953 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.961+0000 7f54b8a45640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54b40b6820 con 0x7f54b40b17d0 2024-03-29T11:25:15.953 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.961+0000 7f54b8a45640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54b40b6f20 con 0x7f54b4142690 2024-03-29T11:25:15.953 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.961+0000 7f54b8a45640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54b414f990 con 0x7f54b41422b0 2024-03-29T11:25:15.953 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.961+0000 7f54b8a45640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:15.954 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.961+0000 7f54b8a45640 10 monclient(hunting): authenticate will time out at 1140.279739s 2024-03-29T11:25:15.954 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.962+0000 7f54b2d9d640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f54b4142690 0x7f54b41e30a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:37470/0 (socket says 172.21.15.86:37470) 2024-03-29T11:25:15.954 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.962+0000 7f54b2d9d640 1 -- 172.21.15.86:0/4212119499 learned_addr learned my addr 172.21.15.86:0/4212119499 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:25:15.954 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.962+0000 7f54a2ffd640 10 client.?.objecter ms_handle_connect 0x7f54b4142690 2024-03-29T11:25:15.954 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.962+0000 7f54a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:15.954 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.962+0000 7f54a2ffd640 1 -- 172.21.15.86:0/4212119499 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3704334402 0 0) 0x7f54b40b6f20 con 0x7f54b4142690 2024-03-29T11:25:15.954 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.962+0000 7f54a2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:15.954 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.962+0000 7f54a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:15.954 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.962+0000 7f54a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:15.955 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.962+0000 7f54a2ffd640 10 monclient(hunting): my global_id is 4372 2024-03-29T11:25:15.955 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.962+0000 7f54a2ffd640 1 -- 172.21.15.86:0/4212119499 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5494004eb0 con 0x7f54b4142690 2024-03-29T11:25:15.955 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.964+0000 7f54a2ffd640 10 client.?.objecter ms_handle_connect 0x7f54b41422b0 2024-03-29T11:25:15.955 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.964+0000 7f54a2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:15.955 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.965+0000 7f54a2ffd640 1 -- 172.21.15.86:0/4212119499 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 578019581 0 0) 0x7f54b414f990 con 0x7f54b41422b0 2024-03-29T11:25:15.955 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.965+0000 7f54a2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:25:15.955 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.965+0000 7f54a2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:15.955 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.965+0000 7f54a2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:15.955 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.965+0000 7f54a2ffd640 10 monclient(hunting): my global_id is 4418 2024-03-29T11:25:15.955 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.965+0000 7f54a2ffd640 1 -- 172.21.15.86:0/4212119499 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54b40b6f20 con 0x7f54b41422b0 2024-03-29T11:25:15.955 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.965+0000 7f54a2ffd640 1 -- 172.21.15.86:0/4212119499 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 522121594 0 0) 0x7f5494004eb0 con 0x7f54b4142690 2024-03-29T11:25:15.955 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.965+0000 7f54a2ffd640 1 -- 172.21.15.86:0/4212119499 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5494005490 con 0x7f54b4142690 2024-03-29T11:25:15.955 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.965+0000 7f54a2ffd640 1 -- 172.21.15.86:0/4212119499 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f54ac001b50 con 0x7f54b4142690 2024-03-29T11:25:15.955 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.965+0000 7f54a2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:15.956 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.965+0000 7f54a2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:15.956 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.965+0000 7f54a2ffd640 10 monclient(hunting): dump: 2024-03-29T11:25:15.956 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:15.956 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:15.956 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:15.956 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:15.956 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:15.956 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:15.956 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:15.956 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:15.956 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:15.956 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:15.956 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.965+0000 7f54a2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:15.956 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.965+0000 7f54a2ffd640 1 -- 172.21.15.86:0/4212119499 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1421998777 0 0) 0x7f54b40b6f20 con 0x7f54b41422b0 2024-03-29T11:25:15.957 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.965+0000 7f54a2ffd640 1 -- 172.21.15.86:0/4212119499 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5494005b60 con 0x7f54b41422b0 2024-03-29T11:25:15.957 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.965+0000 7f54a2ffd640 1 -- 172.21.15.86:0/4212119499 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f54a4001b40 con 0x7f54b41422b0 2024-03-29T11:25:15.957 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.965+0000 7f54a2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:15.957 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.965+0000 7f54a2ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:15.957 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.965+0000 7f54a2ffd640 10 monclient(hunting): dump: 2024-03-29T11:25:15.957 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:15.957 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:15.957 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:15.958 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:15.958 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:15.958 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:15.958 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:15.958 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:15.958 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:15.958 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:15.958 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.966+0000 7f54a2ffd640 1 -- 172.21.15.86:0/4212119499 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 753710380 0 0) 0x7f5494005b60 con 0x7f54b41422b0 2024-03-29T11:25:15.958 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.966+0000 7f54a2ffd640 1 -- 172.21.15.86:0/4212119499 >> v1:172.21.15.196:6789/0 conn(0x7f54b4142690 legacy=0x7f54b41e30a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:15.958 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.966+0000 7f54a2ffd640 1 -- 172.21.15.86:0/4212119499 >> v1:172.21.15.86:6789/0 conn(0x7f54b40b17d0 legacy=0x7f54b414f280 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:15.960 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.968+0000 7f54a2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:25:15.960 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.968+0000 7f54a2ffd640 1 monclient: found mon.c 2024-03-29T11:25:15.960 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.968+0000 7f54a2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:15.960 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.968+0000 7f54a2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:25:15.960 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.968+0000 7f54a2ffd640 1 -- 172.21.15.86:0/4212119499 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b4151000 con 0x7f54b41422b0 2024-03-29T11:25:15.960 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.968+0000 7f54a2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:25:15.961 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.968+0000 7f54a2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:25:15.961 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.968+0000 7f54a2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:15.961 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.968+0000 7f54b8a45640 5 monclient: authenticate success, global_id 4418 2024-03-29T11:25:15.961 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.968+0000 7f54b8a45640 10 monclient: _renew_subs 2024-03-29T11:25:15.961 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.968+0000 7f54b8a45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:25:15.961 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.968+0000 7f54b8a45640 1 -- 172.21.15.86:0/4212119499 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54b4150560 con 0x7f54b41422b0 2024-03-29T11:25:15.961 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.968+0000 7f54b8a45640 10 client.4418.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:15.961 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.968+0000 7f54b8a45640 10 monclient: _renew_subs 2024-03-29T11:25:15.961 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.968+0000 7f54b8a45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:25:15.961 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.968+0000 7f54b8a45640 1 -- 172.21.15.86:0/4212119499 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f54b40b5690 con 0x7f54b41422b0 2024-03-29T11:25:15.961 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.968+0000 7f54b8a45640 1 librados: init done 2024-03-29T11:25:15.961 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.968+0000 7f54a2ffd640 1 -- 172.21.15.86:0/4212119499 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f54a4001d60 con 0x7f54b41422b0 2024-03-29T11:25:15.961 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.968+0000 7f54a2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:15.961 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.968+0000 7f54a2ffd640 1 -- 172.21.15.86:0/4212119499 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f54a4003a40 con 0x7f54b41422b0 2024-03-29T11:25:15.962 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.968+0000 7f54a2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:15.962 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.968+0000 7f54a2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:15.962 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.968+0000 7f54a2ffd640 10 monclient: dump: 2024-03-29T11:25:15.962 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:15.962 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:15.962 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:15.962 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:15.962 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:15.962 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:15.962 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:15.962 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:15.962 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:15.962 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:15.962 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.968+0000 7f54b0d99640 4 set_mon_vals no callback set 2024-03-29T11:25:15.963 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.972+0000 7f54b8a45640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:15.963 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.972+0000 7f54b8a45640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:15.963 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.972+0000 7f54b8a45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:25:15.963 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.972+0000 7f54b8a45640 1 -- 172.21.15.86:0/4212119499 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5498005080 con 0x7f54b41422b0 2024-03-29T11:25:15.963 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.973+0000 7f54a2ffd640 1 -- 172.21.15.86:0/4212119499 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f54a4019a10 con 0x7f54b41422b0 2024-03-29T11:25:15.963 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.974+0000 7f54a2ffd640 1 -- 172.21.15.86:0/4212119499 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f54a4083800 con 0x7f54b41422b0 2024-03-29T11:25:15.963 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.974+0000 7f54a2ffd640 10 client.4418.objecter ms_dispatch 0x7f54b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:15.963 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.974+0000 7f54a2ffd640 3 client.4418.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:15.963 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.974+0000 7f54a2ffd640 3 client.4418.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:15.963 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.974+0000 7f54a2ffd640 20 client.4418.objecter dump_active .. 0 homeless 2024-03-29T11:25:15.963 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.974+0000 7f54b259c640 10 monclient: get_auth_request con 0x7f549406e3d0 auth_method 0 2024-03-29T11:25:15.964 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.975+0000 7f54a2ffd640 10 client.4418.objecter ms_handle_connect 0x7f549406e3d0 2024-03-29T11:25:15.964 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.981+0000 7f54a2ffd640 1 -- 172.21.15.86:0/4212119499 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f54a404f0e0 con 0x7f54b41422b0 2024-03-29T11:25:15.964 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.981+0000 7f54a2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:15.965 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:15.981+0000 7f54a2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:16.262 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:16.277+0000 7f54b8a45640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:16.262 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:16.277+0000 7f54b8a45640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:16.262 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:16.277+0000 7f54b8a45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:25:16.262 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:16.277+0000 7f54b8a45640 1 -- 172.21.15.86:0/4212119499 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5498005330 con 0x7f54b41422b0 2024-03-29T11:25:16.262 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:16.279+0000 7f54a2ffd640 1 -- 172.21.15.86:0/4212119499 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f54a404f0e0 con 0x7f54b41422b0 2024-03-29T11:25:16.263 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:16.279+0000 7f54a2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:16.263 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:16.279+0000 7f54a2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:16.263 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:25:16.263 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:16.266 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:16.282+0000 7f54b8a45640 10 librados: watch_flush enter 2024-03-29T11:25:16.266 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:16.283+0000 7f54b8a45640 10 librados: watch_flush exit 2024-03-29T11:25:16.266 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:16.283+0000 7f54b8a45640 20 client.4418.objecter shutdown clearing up homeless session... 2024-03-29T11:25:16.267 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:16.283+0000 7f54b8a45640 10 client.4418.objecter successfully canceled tick 2024-03-29T11:25:16.267 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:16.283+0000 7f54b8a45640 1 -- 172.21.15.86:0/4212119499 >> v1:172.21.15.196:6816/588110318 conn(0x7f549406e3d0 legacy=0x7f5494070890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:16.267 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:16.283+0000 7f54b8a45640 10 monclient: shutdown 2024-03-29T11:25:16.267 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:16.283+0000 7f54b8a45640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:16.267 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:16.283+0000 7f54b8a45640 1 -- 172.21.15.86:0/4212119499 >> v1:172.21.15.86:6790/0 conn(0x7f54b41422b0 legacy=0x7f54b412dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:16.267 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:16.284+0000 7f54b8a45640 1 -- 172.21.15.86:0/4212119499 shutdown_connections 2024-03-29T11:25:16.268 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:16.284+0000 7f54b8a45640 1 -- 172.21.15.86:0/4212119499 >> 172.21.15.86:0/4212119499 conn(0x7f54b4066a00 msgr2=0x7f54b40b26d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:16.268 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:16.284+0000 7f54b8a45640 1 -- 172.21.15.86:0/4212119499 shutdown_connections 2024-03-29T11:25:16.268 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:16.284+0000 7f54b8a45640 1 -- 172.21.15.86:0/4212119499 wait complete. 2024-03-29T11:25:16.268 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:16.284+0000 7f54b8a45640 1 librados: shutdown 2024-03-29T11:25:16.282 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:23.088 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:23.247 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.263+0000 7f5d4e012640 10 monclient: get_monmap_and_config 2024-03-29T11:25:23.247 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.263+0000 7f5d4e012640 10 monclient: build_initial_monmap 2024-03-29T11:25:23.247 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.263+0000 7f5d4e012640 10 monclient: monmap: 2024-03-29T11:25:23.248 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:25:23.248 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:23.248 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:25:23.264635+0000 2024-03-29T11:25:23.248 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:25:23.264635+0000 2024-03-29T11:25:23.248 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:23.248 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:23.248 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:25:23.248 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:25:23.248 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:25:23.248 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:23.249 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.264+0000 7f5d4e012640 1 Processor -- start 2024-03-29T11:25:23.249 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.264+0000 7f5d4e012640 1 -- start start 2024-03-29T11:25:23.249 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.264+0000 7f5d4e012640 10 monclient: init 2024-03-29T11:25:23.249 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.265+0000 7f5d4e012640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:23.249 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.265+0000 7f5d4e012640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:25:23.249 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.266+0000 7f5d4e012640 10 monclient(hunting): picked mon.noname-c con 0x7f5d480b2800 addr v1:172.21.15.196:6789/0 2024-03-29T11:25:23.252 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.266+0000 7f5d4e012640 10 monclient(hunting): picked mon.noname-b con 0x7f5d480a9c00 addr v1:172.21.15.86:6790/0 2024-03-29T11:25:23.252 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.266+0000 7f5d4d010640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f5d480b2800 0x7f5d48142620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:37754/0 (socket says 172.21.15.86:37754) 2024-03-29T11:25:23.253 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.268+0000 7f5d4e012640 10 monclient(hunting): picked mon.noname-a con 0x7f5d480b38c0 addr v1:172.21.15.86:6789/0 2024-03-29T11:25:23.253 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.268+0000 7f5d4e012640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d48141eb0 con 0x7f5d480b38c0 2024-03-29T11:25:23.253 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.268+0000 7f5d4e012640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d480b22d0 con 0x7f5d480b2800 2024-03-29T11:25:23.253 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.268+0000 7f5d4e012640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d48142d30 con 0x7f5d480a9c00 2024-03-29T11:25:23.253 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.268+0000 7f5d4e012640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:23.253 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.268+0000 7f5d4e012640 10 monclient(hunting): authenticate will time out at 1147.586828s 2024-03-29T11:25:23.253 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.268+0000 7f5d4c80f640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f5d480a9c00 0x7f5d4813bb60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:60537/0 (socket says 172.21.15.86:60537) 2024-03-29T11:25:23.253 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.269+0000 7f5d4c80f640 1 -- 172.21.15.86:0/705063075 learned_addr learned my addr 172.21.15.86:0/705063075 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:25:23.254 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.270+0000 7f5d3ffff640 1 -- 172.21.15.86:0/705063075 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4164241873 0 0) 0x7f5d480b22d0 con 0x7f5d480b2800 2024-03-29T11:25:23.254 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.270+0000 7f5d3ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:23.254 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.270+0000 7f5d3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:23.254 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.270+0000 7f5d3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:23.255 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.270+0000 7f5d3ffff640 10 monclient(hunting): my global_id is 4375 2024-03-29T11:25:23.255 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.270+0000 7f5d3ffff640 1 -- 172.21.15.86:0/705063075 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d34003520 con 0x7f5d480b2800 2024-03-29T11:25:23.257 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.272+0000 7f5d3ffff640 1 -- 172.21.15.86:0/705063075 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1065727078 0 0) 0x7f5d34003520 con 0x7f5d480b2800 2024-03-29T11:25:23.257 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.272+0000 7f5d3ffff640 1 -- 172.21.15.86:0/705063075 >> v1:172.21.15.86:6790/0 conn(0x7f5d480a9c00 legacy=0x7f5d4813bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:23.257 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.272+0000 7f5d3ffff640 1 -- 172.21.15.86:0/705063075 >> v1:172.21.15.86:6789/0 conn(0x7f5d480b38c0 legacy=0x7f5d4814d7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:23.257 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.272+0000 7f5d3ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:25:23.257 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.272+0000 7f5d3ffff640 1 monclient: found mon.noname-c 2024-03-29T11:25:23.258 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.272+0000 7f5d3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:23.258 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.272+0000 7f5d3ffff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T11:25:23.258 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.272+0000 7f5d3ffff640 1 -- 172.21.15.86:0/705063075 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d480aaf90 con 0x7f5d480b2800 2024-03-29T11:25:23.258 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.272+0000 7f5d3ffff640 10 monclient: _finish_auth 0 2024-03-29T11:25:23.258 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.272+0000 7f5d3ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:25:23.258 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.272+0000 7f5d3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:23.258 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.272+0000 7f5d3ffff640 1 -- 172.21.15.86:0/705063075 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f5d440015c0 con 0x7f5d480b2800 2024-03-29T11:25:23.258 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.272+0000 7f5d4e012640 5 monclient: authenticate success, global_id 4375 2024-03-29T11:25:23.258 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.272+0000 7f5d4e012640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:23.258 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.272+0000 7f5d3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:23.258 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.272+0000 7f5d3ffff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:23.258 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.272+0000 7f5d3ffff640 10 monclient: dump: 2024-03-29T11:25:23.258 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:23.258 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:23.259 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:23.259 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:23.259 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:23.259 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:23.259 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:23.259 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:23.259 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:23.259 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:23.259 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.272+0000 7f5d4e012640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:23.259 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.273+0000 7f5d3ffff640 1 -- 172.21.15.86:0/705063075 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d440017e0 con 0x7f5d480b2800 2024-03-29T11:25:23.259 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.273+0000 7f5d3ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:23.259 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.273+0000 7f5d3ffff640 1 -- 172.21.15.86:0/705063075 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f5d44001b70 con 0x7f5d480b2800 2024-03-29T11:25:23.259 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.273+0000 7f5d3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:23.259 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.273+0000 7f5d3ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:23.259 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.273+0000 7f5d3ffff640 10 monclient: dump: 2024-03-29T11:25:23.260 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:23.260 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:23.260 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:23.260 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:23.260 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:23.260 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:23.260 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:23.260 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:23.260 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:23.260 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:23.260 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.273+0000 7f5d4e012640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:23.260 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.273+0000 7f5d4e012640 4 set_mon_vals no callback set 2024-03-29T11:25:23.261 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.274+0000 7f5d4e012640 10 monclient: shutdown 2024-03-29T11:25:23.261 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.274+0000 7f5d4e012640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:23.261 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.274+0000 7f5d4e012640 1 -- 172.21.15.86:0/705063075 >> v1:172.21.15.196:6789/0 conn(0x7f5d480b2800 legacy=0x7f5d48142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:23.261 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.277+0000 7f5d4e012640 1 -- 172.21.15.86:0/705063075 shutdown_connections 2024-03-29T11:25:23.261 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.277+0000 7f5d4e012640 1 -- 172.21.15.86:0/705063075 >> 172.21.15.86:0/705063075 conn(0x7f5d48066a00 msgr2=0x7f5d48066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:23.261 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.278+0000 7f5d4e012640 1 -- 172.21.15.86:0/705063075 shutdown_connections 2024-03-29T11:25:23.261 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.278+0000 7f5d4e012640 1 -- 172.21.15.86:0/705063075 wait complete. 2024-03-29T11:25:23.263 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.278+0000 7f5d4e012640 10 monclient: build_initial_monmap 2024-03-29T11:25:23.263 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.278+0000 7f5d4e012640 10 monclient: monmap: 2024-03-29T11:25:23.263 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:25:23.263 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:23.263 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:25:23.263 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:25:23.263 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:23.264 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:23.264 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:25:23.264 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:25:23.264 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:25:23.264 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:23.264 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.279+0000 7f5d4e012640 1 librados: starting msgr at 2024-03-29T11:25:23.264 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.279+0000 7f5d4e012640 1 librados: starting objecter 2024-03-29T11:25:23.264 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.279+0000 7f5d4e012640 1 Processor -- start 2024-03-29T11:25:23.264 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.279+0000 7f5d4e012640 1 -- start start 2024-03-29T11:25:23.264 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.279+0000 7f5d4e012640 1 librados: setting wanted keys 2024-03-29T11:25:23.264 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.279+0000 7f5d4e012640 1 librados: calling monclient init 2024-03-29T11:25:23.264 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.279+0000 7f5d4e012640 10 monclient: init 2024-03-29T11:25:23.265 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.280+0000 7f5d4e012640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:23.265 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.280+0000 7f5d4e012640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:23.265 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.282+0000 7f5d4e012640 10 monclient(hunting): picked mon.noname-a con 0x7f5d480a9c00 addr v1:172.21.15.86:6789/0 2024-03-29T11:25:23.265 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.282+0000 7f5d4e012640 10 monclient(hunting): picked mon.noname-c con 0x7f5d480b2800 addr v1:172.21.15.86:6790/0 2024-03-29T11:25:23.265 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.282+0000 7f5d4e012640 10 monclient(hunting): picked mon.noname-b con 0x7f5d480b38c0 addr v1:172.21.15.196:6789/0 2024-03-29T11:25:23.265 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.282+0000 7f5d4e012640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d4813aec0 con 0x7f5d480a9c00 2024-03-29T11:25:23.266 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.282+0000 7f5d4e012640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d48070b20 con 0x7f5d480b38c0 2024-03-29T11:25:23.266 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.282+0000 7f5d4e012640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d48071040 con 0x7f5d480b2800 2024-03-29T11:25:23.266 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.282+0000 7f5d4e012640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:23.266 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.282+0000 7f5d4e012640 10 monclient(hunting): authenticate will time out at 1147.600170s 2024-03-29T11:25:23.268 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.284+0000 7f5d4d010640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f5d480a9c00 0x7f5d481de690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:36023/0 (socket says 172.21.15.86:36023) 2024-03-29T11:25:23.268 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.284+0000 7f5d4d010640 1 -- 172.21.15.86:0/901006711 learned_addr learned my addr 172.21.15.86:0/901006711 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:25:23.268 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.284+0000 7f5d3dffb640 10 client.?.objecter ms_handle_connect 0x7f5d480a9c00 2024-03-29T11:25:23.268 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.284+0000 7f5d3dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:23.268 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.284+0000 7f5d3dffb640 10 client.?.objecter ms_handle_connect 0x7f5d480b38c0 2024-03-29T11:25:23.269 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.284+0000 7f5d3dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:23.269 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.284+0000 7f5d3dffb640 10 client.?.objecter ms_handle_connect 0x7f5d480b2800 2024-03-29T11:25:23.269 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.284+0000 7f5d3dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:23.269 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.284+0000 7f5d3dffb640 1 -- 172.21.15.86:0/901006711 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3777133730 0 0) 0x7f5d4813aec0 con 0x7f5d480a9c00 2024-03-29T11:25:23.269 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.284+0000 7f5d3dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:23.269 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.284+0000 7f5d3dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:23.269 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.284+0000 7f5d3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:23.269 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.284+0000 7f5d3dffb640 10 monclient(hunting): my global_id is 4398 2024-03-29T11:25:23.269 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.284+0000 7f5d3dffb640 1 -- 172.21.15.86:0/901006711 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d2c004eb0 con 0x7f5d480a9c00 2024-03-29T11:25:23.269 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.285+0000 7f5d3dffb640 1 -- 172.21.15.86:0/901006711 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2911075483 0 0) 0x7f5d48071040 con 0x7f5d480b2800 2024-03-29T11:25:23.269 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.285+0000 7f5d3dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:23.269 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.285+0000 7f5d3dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:23.269 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.285+0000 7f5d3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:23.269 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.285+0000 7f5d3dffb640 10 monclient(hunting): my global_id is 4424 2024-03-29T11:25:23.270 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.285+0000 7f5d3dffb640 1 -- 172.21.15.86:0/901006711 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d4813aec0 con 0x7f5d480b2800 2024-03-29T11:25:23.270 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.285+0000 7f5d3dffb640 1 -- 172.21.15.86:0/901006711 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3830701198 0 0) 0x7f5d48070b20 con 0x7f5d480b38c0 2024-03-29T11:25:23.270 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.285+0000 7f5d3dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:23.270 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.285+0000 7f5d3dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:23.270 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.285+0000 7f5d3dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:23.270 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.285+0000 7f5d3dffb640 10 monclient(hunting): my global_id is 4378 2024-03-29T11:25:23.270 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.285+0000 7f5d3dffb640 1 -- 172.21.15.86:0/901006711 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5d48071040 con 0x7f5d480b38c0 2024-03-29T11:25:23.272 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.285+0000 7f5d3dffb640 1 -- 172.21.15.86:0/901006711 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4116936621 0 0) 0x7f5d2c004eb0 con 0x7f5d480a9c00 2024-03-29T11:25:23.272 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.285+0000 7f5d3dffb640 1 -- 172.21.15.86:0/901006711 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d48070b20 con 0x7f5d480a9c00 2024-03-29T11:25:23.272 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.285+0000 7f5d3dffb640 1 -- 172.21.15.86:0/901006711 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f5d44001540 con 0x7f5d480a9c00 2024-03-29T11:25:23.272 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.285+0000 7f5d3dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:23.272 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.285+0000 7f5d3dffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:23.272 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.285+0000 7f5d3dffb640 10 monclient(hunting): dump: 2024-03-29T11:25:23.273 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:23.273 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:23.273 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:23.273 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:23.273 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:23.273 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:23.273 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:23.273 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:23.273 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:23.273 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:23.273 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.285+0000 7f5d3dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:23.273 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.285+0000 7f5d3dffb640 1 -- 172.21.15.86:0/901006711 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2598771318 0 0) 0x7f5d4813aec0 con 0x7f5d480b2800 2024-03-29T11:25:23.273 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.285+0000 7f5d3dffb640 1 -- 172.21.15.86:0/901006711 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d2c004eb0 con 0x7f5d480b2800 2024-03-29T11:25:23.273 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.286+0000 7f5d3dffb640 1 -- 172.21.15.86:0/901006711 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f5d380018b0 con 0x7f5d480b2800 2024-03-29T11:25:23.274 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.286+0000 7f5d3dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:23.274 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.286+0000 7f5d3dffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:23.274 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.286+0000 7f5d3dffb640 10 monclient(hunting): dump: 2024-03-29T11:25:23.274 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:23.274 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:23.274 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:23.274 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:23.274 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:23.274 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:23.274 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:23.274 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:23.274 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:23.274 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:23.274 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.286+0000 7f5d3dffb640 1 -- 172.21.15.86:0/901006711 <== mon.0 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2791542703 0 0) 0x7f5d48070b20 con 0x7f5d480a9c00 2024-03-29T11:25:23.275 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.286+0000 7f5d3dffb640 1 -- 172.21.15.86:0/901006711 >> v1:172.21.15.86:6790/0 conn(0x7f5d480b2800 legacy=0x7f5d4813a0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:23.275 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.286+0000 7f5d3dffb640 1 -- 172.21.15.86:0/901006711 >> v1:172.21.15.196:6789/0 conn(0x7f5d480b38c0 legacy=0x7f5d4813a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:23.275 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.286+0000 7f5d3dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:25:23.275 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.286+0000 7f5d3dffb640 1 monclient: found mon.a 2024-03-29T11:25:23.275 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.286+0000 7f5d3dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:23.275 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.286+0000 7f5d3dffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:25:23.275 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.286+0000 7f5d3dffb640 1 -- 172.21.15.86:0/901006711 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d48070600 con 0x7f5d480a9c00 2024-03-29T11:25:23.275 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.286+0000 7f5d3dffb640 10 monclient: _finish_auth 0 2024-03-29T11:25:23.275 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.286+0000 7f5d3dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:25:23.275 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.286+0000 7f5d3dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:23.275 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.286+0000 7f5d4e012640 5 monclient: authenticate success, global_id 4398 2024-03-29T11:25:23.276 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.286+0000 7f5d4e012640 10 monclient: _renew_subs 2024-03-29T11:25:23.276 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.286+0000 7f5d4e012640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:25:23.276 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.286+0000 7f5d4e012640 1 -- 172.21.15.86:0/901006711 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d48070850 con 0x7f5d480a9c00 2024-03-29T11:25:23.276 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.286+0000 7f5d3dffb640 1 -- 172.21.15.86:0/901006711 <== mon.0 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d44004a90 con 0x7f5d480a9c00 2024-03-29T11:25:23.276 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.286+0000 7f5d3dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:23.276 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.286+0000 7f5d3dffb640 1 -- 172.21.15.86:0/901006711 <== mon.0 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f5d44004da0 con 0x7f5d480a9c00 2024-03-29T11:25:23.276 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.286+0000 7f5d3dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:23.276 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.286+0000 7f5d3dffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:23.276 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.286+0000 7f5d3dffb640 10 monclient: dump: 2024-03-29T11:25:23.276 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:23.276 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:23.276 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:23.276 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:23.276 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:23.277 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:23.277 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:23.277 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:23.277 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:23.277 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:23.277 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.286+0000 7f5d3f7fe640 4 set_mon_vals no callback set 2024-03-29T11:25:23.277 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.287+0000 7f5d4e012640 10 client.4398.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:23.277 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.287+0000 7f5d4e012640 10 monclient: _renew_subs 2024-03-29T11:25:23.277 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.287+0000 7f5d4e012640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:25:23.277 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.287+0000 7f5d4e012640 1 -- 172.21.15.86:0/901006711 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d4813eee0 con 0x7f5d480a9c00 2024-03-29T11:25:23.277 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.287+0000 7f5d4e012640 1 librados: init done 2024-03-29T11:25:23.277 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.288+0000 7f5d337fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:23.277 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.288+0000 7f5d337fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:23.277 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.288+0000 7f5d337fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:25:23.277 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.288+0000 7f5d337fe640 1 -- 172.21.15.86:0/901006711 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d48070d60 con 0x7f5d480a9c00 2024-03-29T11:25:23.278 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.289+0000 7f5d3dffb640 1 -- 172.21.15.86:0/901006711 <== mon.0 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f5d4401ad70 con 0x7f5d480a9c00 2024-03-29T11:25:23.278 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.290+0000 7f5d4c80f640 10 monclient: get_auth_request con 0x7f5d2c06de20 auth_method 0 2024-03-29T11:25:23.278 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.290+0000 7f5d3dffb640 1 -- 172.21.15.86:0/901006711 <== mon.0 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f5d44054990 con 0x7f5d480a9c00 2024-03-29T11:25:23.278 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.290+0000 7f5d3dffb640 10 client.4398.objecter ms_dispatch 0x7f5d48000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:23.278 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.290+0000 7f5d3dffb640 3 client.4398.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:23.278 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.290+0000 7f5d3dffb640 3 client.4398.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:23.278 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.290+0000 7f5d3dffb640 20 client.4398.objecter dump_active .. 0 homeless 2024-03-29T11:25:23.278 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.291+0000 7f5d3dffb640 10 client.4398.objecter ms_handle_connect 0x7f5d2c06de20 2024-03-29T11:25:23.284 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.299+0000 7f5d3dffb640 1 -- 172.21.15.86:0/901006711 <== mon.0 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5d48070d60 con 0x7f5d480a9c00 2024-03-29T11:25:23.284 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.299+0000 7f5d3dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:23.284 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.299+0000 7f5d3dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:23.616 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.632+0000 7f5d337fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:23.616 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.632+0000 7f5d337fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:23.617 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.632+0000 7f5d337fe640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:25:23.617 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.632+0000 7f5d337fe640 1 -- 172.21.15.86:0/901006711 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d4804f730 con 0x7f5d480a9c00 2024-03-29T11:25:23.618 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:25:23.619 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:23.619 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.633+0000 7f5d3dffb640 1 -- 172.21.15.86:0/901006711 <== mon.0 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f5d44054060 con 0x7f5d480a9c00 2024-03-29T11:25:23.619 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.633+0000 7f5d3dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:23.619 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.633+0000 7f5d3dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:23.620 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.636+0000 7f5d337fe640 10 librados: watch_flush enter 2024-03-29T11:25:23.621 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.637+0000 7f5d337fe640 10 librados: watch_flush exit 2024-03-29T11:25:23.622 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.637+0000 7f5d337fe640 20 client.4398.objecter shutdown clearing up homeless session... 2024-03-29T11:25:23.622 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.637+0000 7f5d337fe640 10 client.4398.objecter successfully canceled tick 2024-03-29T11:25:23.622 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.637+0000 7f5d337fe640 1 -- 172.21.15.86:0/901006711 >> v1:172.21.15.196:6816/588110318 conn(0x7f5d2c06de20 legacy=0x7f5d2c0702e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:23.622 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.637+0000 7f5d337fe640 10 monclient: shutdown 2024-03-29T11:25:23.622 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.637+0000 7f5d337fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:23.622 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.637+0000 7f5d337fe640 1 -- 172.21.15.86:0/901006711 >> v1:172.21.15.86:6789/0 conn(0x7f5d480a9c00 legacy=0x7f5d481de690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:23.622 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.638+0000 7f5d337fe640 1 -- 172.21.15.86:0/901006711 shutdown_connections 2024-03-29T11:25:23.622 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.638+0000 7f5d337fe640 1 -- 172.21.15.86:0/901006711 >> 172.21.15.86:0/901006711 conn(0x7f5d48066a00 msgr2=0x7f5d4813bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:23.622 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.638+0000 7f5d337fe640 1 -- 172.21.15.86:0/901006711 shutdown_connections 2024-03-29T11:25:23.622 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.638+0000 7f5d337fe640 1 -- 172.21.15.86:0/901006711 wait complete. 2024-03-29T11:25:23.623 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:23.638+0000 7f5d337fe640 1 librados: shutdown 2024-03-29T11:25:23.637 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:30.444 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:30.575 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.591+0000 7f9eba20f640 10 monclient: get_monmap_and_config 2024-03-29T11:25:30.575 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.591+0000 7f9eba20f640 10 monclient: build_initial_monmap 2024-03-29T11:25:30.576 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.591+0000 7f9eba20f640 10 monclient: monmap: 2024-03-29T11:25:30.576 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:25:30.576 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:30.576 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:25:30.592660+0000 2024-03-29T11:25:30.576 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:25:30.592660+0000 2024-03-29T11:25:30.576 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:30.576 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:30.576 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:25:30.576 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:25:30.576 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:25:30.576 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:30.577 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.592+0000 7f9eba20f640 1 Processor -- start 2024-03-29T11:25:30.577 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.592+0000 7f9eba20f640 1 -- start start 2024-03-29T11:25:30.577 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.592+0000 7f9eba20f640 10 monclient: init 2024-03-29T11:25:30.577 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.593+0000 7f9eba20f640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:30.577 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.593+0000 7f9eba20f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:25:30.577 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.593+0000 7f9eba20f640 10 monclient(hunting): picked mon.noname-b con 0x7f9eb41422b0 addr v1:172.21.15.86:6790/0 2024-03-29T11:25:30.578 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.593+0000 7f9eba20f640 10 monclient(hunting): picked mon.noname-a con 0x7f9eb4142690 addr v1:172.21.15.86:6789/0 2024-03-29T11:25:30.578 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.593+0000 7f9eba20f640 10 monclient(hunting): picked mon.noname-c con 0x7f9eb40b17d0 addr v1:172.21.15.196:6789/0 2024-03-29T11:25:30.578 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.593+0000 7f9eba20f640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eb4142c80 con 0x7f9eb4142690 2024-03-29T11:25:30.578 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.593+0000 7f9eba20f640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eb40aae40 con 0x7f9eb40b17d0 2024-03-29T11:25:30.578 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.593+0000 7f9eba20f640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eb413fbd0 con 0x7f9eb41422b0 2024-03-29T11:25:30.578 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.593+0000 7f9eba20f640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:30.578 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.593+0000 7f9eba20f640 10 monclient(hunting): authenticate will time out at 1154.911704s 2024-03-29T11:25:30.579 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.594+0000 7f9eb37fe640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f9eb41422b0 0x7f9eb413f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:59770/0 (socket says 172.21.15.86:59770) 2024-03-29T11:25:30.579 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.594+0000 7f9eb37fe640 1 -- 172.21.15.86:0/3408901426 learned_addr learned my addr 172.21.15.86:0/3408901426 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:25:30.579 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.594+0000 7f9eb27fc640 1 -- 172.21.15.86:0/3408901426 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1341274586 0 0) 0x7f9eb413fbd0 con 0x7f9eb41422b0 2024-03-29T11:25:30.579 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.594+0000 7f9eb27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:30.579 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.594+0000 7f9eb27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:30.579 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.594+0000 7f9eb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:30.579 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.594+0000 7f9eb27fc640 10 monclient(hunting): my global_id is 4427 2024-03-29T11:25:30.579 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.594+0000 7f9eb27fc640 1 -- 172.21.15.86:0/3408901426 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e9c003520 con 0x7f9eb41422b0 2024-03-29T11:25:30.579 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.594+0000 7f9eb27fc640 1 -- 172.21.15.86:0/3408901426 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3019303558 0 0) 0x7f9eb40aae40 con 0x7f9eb40b17d0 2024-03-29T11:25:30.579 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.594+0000 7f9eb27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:30.579 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.594+0000 7f9eb27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:30.580 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.594+0000 7f9eb27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:30.580 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.594+0000 7f9eb27fc640 10 monclient(hunting): my global_id is 4381 2024-03-29T11:25:30.580 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.594+0000 7f9eb27fc640 1 -- 172.21.15.86:0/3408901426 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9eb413fbd0 con 0x7f9eb40b17d0 2024-03-29T11:25:30.580 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.594+0000 7f9eb27fc640 1 -- 172.21.15.86:0/3408901426 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4181304767 0 0) 0x7f9e9c003520 con 0x7f9eb41422b0 2024-03-29T11:25:30.580 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.594+0000 7f9eb27fc640 1 -- 172.21.15.86:0/3408901426 >> v1:172.21.15.196:6789/0 conn(0x7f9eb40b17d0 legacy=0x7f9eb40b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:30.580 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.594+0000 7f9eb27fc640 1 -- 172.21.15.86:0/3408901426 >> v1:172.21.15.86:6789/0 conn(0x7f9eb4142690 legacy=0x7f9eb40b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:30.581 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.596+0000 7f9eb27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:25:30.581 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.596+0000 7f9eb27fc640 1 monclient: found mon.noname-b 2024-03-29T11:25:30.581 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.596+0000 7f9eb27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:30.581 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.596+0000 7f9eb27fc640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:25:30.581 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.596+0000 7f9eb27fc640 1 -- 172.21.15.86:0/3408901426 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb4151940 con 0x7f9eb41422b0 2024-03-29T11:25:30.581 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.596+0000 7f9eb27fc640 10 monclient: _finish_auth 0 2024-03-29T11:25:30.581 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.596+0000 7f9eb27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:25:30.581 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.596+0000 7f9eb27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:30.581 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.596+0000 7f9eba20f640 5 monclient: authenticate success, global_id 4427 2024-03-29T11:25:30.581 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.596+0000 7f9eb27fc640 1 -- 172.21.15.86:0/3408901426 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f9ea4001690 con 0x7f9eb41422b0 2024-03-29T11:25:30.582 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.596+0000 7f9eb27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:30.582 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.596+0000 7f9eb27fc640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:30.582 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.596+0000 7f9eb27fc640 10 monclient: dump: 2024-03-29T11:25:30.582 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:30.582 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:30.582 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:30.582 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:30.582 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:30.582 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:30.582 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:30.582 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:30.582 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:30.582 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:30.582 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.596+0000 7f9eba20f640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:30.583 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.596+0000 7f9eb27fc640 1 -- 172.21.15.86:0/3408901426 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ea40017f0 con 0x7f9eb41422b0 2024-03-29T11:25:30.583 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.596+0000 7f9eb27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:30.583 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.596+0000 7f9eb27fc640 1 -- 172.21.15.86:0/3408901426 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f9ea4001b80 con 0x7f9eb41422b0 2024-03-29T11:25:30.583 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.596+0000 7f9eb27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:30.583 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.596+0000 7f9eb27fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:30.583 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.596+0000 7f9eb27fc640 10 monclient: dump: 2024-03-29T11:25:30.583 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:30.583 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:30.583 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:30.583 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:30.583 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:30.583 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:30.583 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:30.583 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:30.584 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:30.584 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:30.584 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.596+0000 7f9eba20f640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:30.584 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.596+0000 7f9eba20f640 4 set_mon_vals no callback set 2024-03-29T11:25:30.584 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.597+0000 7f9eba20f640 10 monclient: shutdown 2024-03-29T11:25:30.589 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.597+0000 7f9eba20f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:30.590 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.597+0000 7f9eba20f640 1 -- 172.21.15.86:0/3408901426 >> v1:172.21.15.86:6790/0 conn(0x7f9eb41422b0 legacy=0x7f9eb413f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:30.590 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.597+0000 7f9eba20f640 1 -- 172.21.15.86:0/3408901426 shutdown_connections 2024-03-29T11:25:30.590 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.597+0000 7f9eba20f640 1 -- 172.21.15.86:0/3408901426 >> 172.21.15.86:0/3408901426 conn(0x7f9eb4066a00 msgr2=0x7f9eb4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:30.590 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.597+0000 7f9eba20f640 1 -- 172.21.15.86:0/3408901426 shutdown_connections 2024-03-29T11:25:30.590 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.597+0000 7f9eba20f640 1 -- 172.21.15.86:0/3408901426 wait complete. 2024-03-29T11:25:30.590 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.598+0000 7f9eba20f640 10 monclient: build_initial_monmap 2024-03-29T11:25:30.590 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.598+0000 7f9eba20f640 10 monclient: monmap: 2024-03-29T11:25:30.590 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:25:30.590 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:30.590 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:25:30.590 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:25:30.590 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:30.590 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:30.590 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:25:30.591 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:25:30.591 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:25:30.591 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:30.591 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.598+0000 7f9eba20f640 1 librados: starting msgr at 2024-03-29T11:25:30.591 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.598+0000 7f9eba20f640 1 librados: starting objecter 2024-03-29T11:25:30.591 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.599+0000 7f9eba20f640 1 Processor -- start 2024-03-29T11:25:30.591 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.599+0000 7f9eba20f640 1 -- start start 2024-03-29T11:25:30.591 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.599+0000 7f9eba20f640 1 librados: setting wanted keys 2024-03-29T11:25:30.591 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.599+0000 7f9eba20f640 1 librados: calling monclient init 2024-03-29T11:25:30.591 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.599+0000 7f9eba20f640 10 monclient: init 2024-03-29T11:25:30.591 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.599+0000 7f9eba20f640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:30.591 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.599+0000 7f9eba20f640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:25:30.591 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.599+0000 7f9eba20f640 10 monclient(hunting): picked mon.noname-a con 0x7f9eb40b17d0 addr v1:172.21.15.86:6789/0 2024-03-29T11:25:30.591 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.599+0000 7f9eba20f640 10 monclient(hunting): picked mon.noname-c con 0x7f9eb41422b0 addr v1:172.21.15.86:6790/0 2024-03-29T11:25:30.592 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.599+0000 7f9eba20f640 10 monclient(hunting): picked mon.noname-b con 0x7f9eb4142690 addr v1:172.21.15.196:6789/0 2024-03-29T11:25:30.592 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.599+0000 7f9eba20f640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eb412c650 con 0x7f9eb40b17d0 2024-03-29T11:25:30.592 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.599+0000 7f9eba20f640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eb412dfd0 con 0x7f9eb4142690 2024-03-29T11:25:30.592 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.599+0000 7f9eba20f640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9eb412e2b0 con 0x7f9eb41422b0 2024-03-29T11:25:30.592 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.599+0000 7f9eba20f640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:30.592 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.599+0000 7f9eba20f640 10 monclient(hunting): authenticate will time out at 1154.917921s 2024-03-29T11:25:30.592 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.600+0000 7f9eb2ffd640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f9eb41422b0 0x7f9eb412b830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:34860/0 (socket says 172.21.15.86:34860) 2024-03-29T11:25:30.592 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.600+0000 7f9eb2ffd640 1 -- 172.21.15.86:0/2456336425 learned_addr learned my addr 172.21.15.86:0/2456336425 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:25:30.592 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.601+0000 7f9e9bfff640 10 client.?.objecter ms_handle_connect 0x7f9eb41422b0 2024-03-29T11:25:30.592 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.601+0000 7f9e9bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:30.592 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.601+0000 7f9e9bfff640 10 client.?.objecter ms_handle_connect 0x7f9eb4142690 2024-03-29T11:25:30.592 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.601+0000 7f9e9bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:30.592 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.601+0000 7f9e9bfff640 1 -- 172.21.15.86:0/2456336425 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1059075549 0 0) 0x7f9eb412e2b0 con 0x7f9eb41422b0 2024-03-29T11:25:30.592 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.601+0000 7f9e9bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:30.593 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.601+0000 7f9e9bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:30.593 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.601+0000 7f9e9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:30.593 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.601+0000 7f9e9bfff640 10 monclient(hunting): my global_id is 4430 2024-03-29T11:25:30.593 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.601+0000 7f9e9bfff640 1 -- 172.21.15.86:0/2456336425 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ea0004eb0 con 0x7f9eb41422b0 2024-03-29T11:25:30.593 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.601+0000 7f9e9bfff640 1 -- 172.21.15.86:0/2456336425 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2847009994 0 0) 0x7f9eb412dfd0 con 0x7f9eb4142690 2024-03-29T11:25:30.593 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.601+0000 7f9e9bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:25:30.593 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.601+0000 7f9e9bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:30.593 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.601+0000 7f9e9bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:30.593 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.601+0000 7f9e9bfff640 10 monclient(hunting): my global_id is 4384 2024-03-29T11:25:30.593 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.601+0000 7f9e9bfff640 1 -- 172.21.15.86:0/2456336425 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ea0006230 con 0x7f9eb4142690 2024-03-29T11:25:30.593 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.602+0000 7f9e9bfff640 1 -- 172.21.15.86:0/2456336425 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3847280016 0 0) 0x7f9ea0006230 con 0x7f9eb4142690 2024-03-29T11:25:30.593 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.602+0000 7f9e9bfff640 1 -- 172.21.15.86:0/2456336425 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9eb412dfd0 con 0x7f9eb4142690 2024-03-29T11:25:30.593 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.602+0000 7f9e9bfff640 1 -- 172.21.15.86:0/2456336425 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f9eac001cb0 con 0x7f9eb4142690 2024-03-29T11:25:30.593 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.602+0000 7f9e9bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:30.593 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.602+0000 7f9e9bfff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:30.594 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.602+0000 7f9e9bfff640 10 monclient(hunting): dump: 2024-03-29T11:25:30.594 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:30.594 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:30.594 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:30.594 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:30.594 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:30.594 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:30.594 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:30.594 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:30.594 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:30.594 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:30.594 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.602+0000 7f9e9bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:30.594 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.603+0000 7f9e9bfff640 1 -- 172.21.15.86:0/2456336425 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1028585662 0 0) 0x7f9ea0004eb0 con 0x7f9eb41422b0 2024-03-29T11:25:30.594 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.603+0000 7f9e9bfff640 1 -- 172.21.15.86:0/2456336425 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9ea0006230 con 0x7f9eb41422b0 2024-03-29T11:25:30.595 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.603+0000 7f9e9bfff640 1 -- 172.21.15.86:0/2456336425 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f9ea8001790 con 0x7f9eb41422b0 2024-03-29T11:25:30.595 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.603+0000 7f9e9bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:30.595 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.603+0000 7f9e9bfff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:30.595 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.603+0000 7f9e9bfff640 10 monclient(hunting): dump: 2024-03-29T11:25:30.595 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:30.595 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:30.595 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:30.595 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:30.595 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:30.595 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:30.595 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:30.595 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:30.595 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:30.595 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:30.596 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.603+0000 7f9e9bfff640 1 -- 172.21.15.86:0/2456336425 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2071247632 0 0) 0x7f9ea0006230 con 0x7f9eb41422b0 2024-03-29T11:25:30.596 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.603+0000 7f9e9bfff640 1 -- 172.21.15.86:0/2456336425 >> v1:172.21.15.196:6789/0 conn(0x7f9eb4142690 legacy=0x7f9eb412bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:30.596 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.603+0000 7f9e9bfff640 1 -- 172.21.15.86:0/2456336425 >> v1:172.21.15.86:6789/0 conn(0x7f9eb40b17d0 legacy=0x7f9eb412fec0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:30.596 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.603+0000 7f9e9bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:25:30.596 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.603+0000 7f9e9bfff640 1 monclient: found mon.c 2024-03-29T11:25:30.596 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.603+0000 7f9e9bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:30.596 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.603+0000 7f9e9bfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:25:30.596 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.603+0000 7f9e9bfff640 1 -- 172.21.15.86:0/2456336425 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9eb412e590 con 0x7f9eb41422b0 2024-03-29T11:25:30.596 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.603+0000 7f9e9bfff640 10 monclient: _finish_auth 0 2024-03-29T11:25:30.596 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.603+0000 7f9e9bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:25:30.596 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.603+0000 7f9e9bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:30.596 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.603+0000 7f9eba20f640 5 monclient: authenticate success, global_id 4430 2024-03-29T11:25:30.596 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.603+0000 7f9eba20f640 10 monclient: _renew_subs 2024-03-29T11:25:30.596 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.603+0000 7f9eba20f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:25:30.596 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.603+0000 7f9eba20f640 1 -- 172.21.15.86:0/2456336425 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eb412ce50 con 0x7f9eb41422b0 2024-03-29T11:25:30.597 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.604+0000 7f9eba20f640 10 client.4430.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:30.597 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.604+0000 7f9eba20f640 10 monclient: _renew_subs 2024-03-29T11:25:30.597 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.604+0000 7f9eba20f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:25:30.597 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.604+0000 7f9eba20f640 1 -- 172.21.15.86:0/2456336425 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eb412f000 con 0x7f9eb41422b0 2024-03-29T11:25:30.597 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.604+0000 7f9eba20f640 1 librados: init done 2024-03-29T11:25:30.597 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.604+0000 7f9e9bfff640 1 -- 172.21.15.86:0/2456336425 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9ea8001ca0 con 0x7f9eb41422b0 2024-03-29T11:25:30.597 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.604+0000 7f9e9bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:30.597 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.604+0000 7f9eb17fa640 4 set_mon_vals no callback set 2024-03-29T11:25:30.597 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.604+0000 7f9e9bfff640 1 -- 172.21.15.86:0/2456336425 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f9ea8004a00 con 0x7f9eb41422b0 2024-03-29T11:25:30.597 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.604+0000 7f9e9bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:30.597 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.604+0000 7f9e9bfff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:30.597 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.604+0000 7f9e9bfff640 10 monclient: dump: 2024-03-29T11:25:30.597 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:30.597 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:30.597 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:30.598 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:30.598 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:30.598 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:30.598 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:30.598 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:30.598 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:30.598 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:30.598 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.605+0000 7f9eba20f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:30.598 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.605+0000 7f9eba20f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:30.598 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.605+0000 7f9eba20f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:25:30.598 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.605+0000 7f9eba20f640 1 -- 172.21.15.86:0/2456336425 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e94005080 con 0x7f9eb41422b0 2024-03-29T11:25:30.599 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.606+0000 7f9e9bfff640 1 -- 172.21.15.86:0/2456336425 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f9ea801a9d0 con 0x7f9eb41422b0 2024-03-29T11:25:30.599 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.607+0000 7f9e9bfff640 1 -- 172.21.15.86:0/2456336425 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f9ea8084970 con 0x7f9eb41422b0 2024-03-29T11:25:30.600 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.607+0000 7f9e9bfff640 10 client.4430.objecter ms_dispatch 0x7f9eb4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:30.600 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.607+0000 7f9e9bfff640 3 client.4430.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:30.600 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.607+0000 7f9e9bfff640 3 client.4430.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:30.600 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.607+0000 7f9e9bfff640 20 client.4430.objecter dump_active .. 0 homeless 2024-03-29T11:25:30.600 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.607+0000 7f9eb37fe640 10 monclient: get_auth_request con 0x7f9ea006dfe0 auth_method 0 2024-03-29T11:25:30.600 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.608+0000 7f9e9bfff640 10 client.4430.objecter ms_handle_connect 0x7f9ea006dfe0 2024-03-29T11:25:30.600 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.614+0000 7f9e9bfff640 1 -- 172.21.15.86:0/2456336425 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9ea80502b0 con 0x7f9eb41422b0 2024-03-29T11:25:30.600 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.614+0000 7f9e9bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:30.600 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.614+0000 7f9e9bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:30.913 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.929+0000 7f9eba20f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:30.913 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.929+0000 7f9eba20f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:30.913 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.929+0000 7f9eba20f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:25:30.913 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.929+0000 7f9eba20f640 1 -- 172.21.15.86:0/2456336425 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e94005330 con 0x7f9eb41422b0 2024-03-29T11:25:30.915 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.931+0000 7f9e9bfff640 1 -- 172.21.15.86:0/2456336425 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f9ea80521e0 con 0x7f9eb41422b0 2024-03-29T11:25:30.915 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.931+0000 7f9e9bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:30.915 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.931+0000 7f9e9bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:30.916 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:25:30.917 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:30.919 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.935+0000 7f9eba20f640 10 librados: watch_flush enter 2024-03-29T11:25:30.919 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.935+0000 7f9eba20f640 10 librados: watch_flush exit 2024-03-29T11:25:30.919 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.935+0000 7f9eba20f640 20 client.4430.objecter shutdown clearing up homeless session... 2024-03-29T11:25:30.919 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.935+0000 7f9eba20f640 10 client.4430.objecter successfully canceled tick 2024-03-29T11:25:30.919 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.935+0000 7f9eba20f640 1 -- 172.21.15.86:0/2456336425 >> v1:172.21.15.196:6816/588110318 conn(0x7f9ea006dfe0 legacy=0x7f9ea00704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:30.919 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.935+0000 7f9eba20f640 10 monclient: shutdown 2024-03-29T11:25:30.919 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.935+0000 7f9eba20f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:30.920 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.935+0000 7f9eba20f640 1 -- 172.21.15.86:0/2456336425 >> v1:172.21.15.86:6790/0 conn(0x7f9eb41422b0 legacy=0x7f9eb412b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:30.920 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.935+0000 7f9eba20f640 1 -- 172.21.15.86:0/2456336425 shutdown_connections 2024-03-29T11:25:30.920 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.935+0000 7f9eba20f640 1 -- 172.21.15.86:0/2456336425 >> 172.21.15.86:0/2456336425 conn(0x7f9eb4066a00 msgr2=0x7f9eb413db20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:30.920 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.935+0000 7f9eba20f640 1 -- 172.21.15.86:0/2456336425 shutdown_connections 2024-03-29T11:25:30.921 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.935+0000 7f9eba20f640 1 -- 172.21.15.86:0/2456336425 wait complete. 2024-03-29T11:25:30.921 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:30.936+0000 7f9eba20f640 1 librados: shutdown 2024-03-29T11:25:30.939 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:37.746 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:37.900 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.916+0000 7facd7fd6640 10 monclient: get_monmap_and_config 2024-03-29T11:25:37.900 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.916+0000 7facd7fd6640 10 monclient: build_initial_monmap 2024-03-29T11:25:37.900 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.916+0000 7facd7fd6640 10 monclient: monmap: 2024-03-29T11:25:37.900 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:25:37.901 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:37.901 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:25:37.917015+0000 2024-03-29T11:25:37.901 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:25:37.917015+0000 2024-03-29T11:25:37.901 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:37.901 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:37.901 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:25:37.901 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:25:37.901 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:25:37.901 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:37.903 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.918+0000 7facd7fd6640 1 Processor -- start 2024-03-29T11:25:37.906 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.919+0000 7facd7fd6640 1 -- start start 2024-03-29T11:25:37.906 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.919+0000 7facd7fd6640 10 monclient: init 2024-03-29T11:25:37.906 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.920+0000 7facd7fd6640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:37.906 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.920+0000 7facd7fd6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:25:37.906 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.920+0000 7facd7fd6640 10 monclient(hunting): picked mon.noname-b con 0x7facd006ebd0 addr v1:172.21.15.86:6790/0 2024-03-29T11:25:37.906 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.920+0000 7facd7fd6640 10 monclient(hunting): picked mon.noname-c con 0x7facd006efb0 addr v1:172.21.15.196:6789/0 2024-03-29T11:25:37.906 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.920+0000 7facd7fd6640 10 monclient(hunting): picked mon.noname-a con 0x7facd013ed90 addr v1:172.21.15.86:6789/0 2024-03-29T11:25:37.906 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.920+0000 7facd7fd6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd0070b40 con 0x7facd013ed90 2024-03-29T11:25:37.906 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.920+0000 7facd7fd6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd0070e20 con 0x7facd006efb0 2024-03-29T11:25:37.906 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.920+0000 7facd7fd6640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd0071100 con 0x7facd006ebd0 2024-03-29T11:25:37.907 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.920+0000 7facd7fd6640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:37.907 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.920+0000 7facd7fd6640 10 monclient(hunting): authenticate will time out at 1162.238306s 2024-03-29T11:25:37.907 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.923+0000 7facd5d72640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7facd006ebd0 0x7facd013b930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:39875/0 (socket says 172.21.15.86:39875) 2024-03-29T11:25:37.907 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.923+0000 7facd5d72640 1 -- 172.21.15.86:0/3028201641 learned_addr learned my addr 172.21.15.86:0/3028201641 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:25:37.909 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.924+0000 7facd4d70640 1 -- 172.21.15.86:0/3028201641 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1606345973 0 0) 0x7facd0071100 con 0x7facd006ebd0 2024-03-29T11:25:37.909 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.924+0000 7facd4d70640 10 monclient(hunting): _negotiate 2024-03-29T11:25:37.909 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.924+0000 7facd4d70640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:37.909 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.924+0000 7facd4d70640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:37.909 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.924+0000 7facd6573640 1 -- 172.21.15.86:0/3028201641 >> v1:172.21.15.86:6789/0 conn(0x7facd013ed90 legacy=0x7facd0149d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-03-29T11:25:37.909 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.924+0000 7facd6573640 1 -- 172.21.15.86:0/3028201641 >> v1:172.21.15.86:6789/0 conn(0x7facd013ed90 legacy=0x7facd0149d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-29T11:25:37.909 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.924+0000 7facd6573640 1 --1- 172.21.15.86:0/3028201641 >> v1:172.21.15.86:6789/0 conn(0x7facd013ed90 0x7facd0149d50 :-1 s=CONNECTING_SEND_CONNECT_MSG pgs=0 cs=0 l=0).handle_connect_reply_1 read connect reply failed 2024-03-29T11:25:37.909 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.924+0000 7facd4d70640 10 monclient(hunting): my global_id is 4433 2024-03-29T11:25:37.910 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.924+0000 7facd4d70640 1 -- 172.21.15.86:0/3028201641 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facbc003520 con 0x7facd006ebd0 2024-03-29T11:25:37.910 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.925+0000 7facd4d70640 1 -- 172.21.15.86:0/3028201641 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 155110705 0 0) 0x7facbc003520 con 0x7facd006ebd0 2024-03-29T11:25:37.910 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.925+0000 7facd4d70640 1 -- 172.21.15.86:0/3028201641 >> v1:172.21.15.196:6789/0 conn(0x7facd006efb0 legacy=0x7facd013e680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:37.910 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.925+0000 7facd4d70640 1 -- 172.21.15.86:0/3028201641 >> v1:172.21.15.86:6789/0 conn(0x7facd013ed90 legacy=0x7facd0149d50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:25:37.910 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.925+0000 7facd4d70640 10 monclient: _finish_hunting 0 2024-03-29T11:25:37.910 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.926+0000 7facd4d70640 1 monclient: found mon.noname-b 2024-03-29T11:25:37.910 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.926+0000 7facd4d70640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:37.910 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.926+0000 7facd4d70640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:25:37.910 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.926+0000 7facd4d70640 1 -- 172.21.15.86:0/3028201641 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd013c040 con 0x7facd006ebd0 2024-03-29T11:25:37.911 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.926+0000 7facd4d70640 10 monclient: _finish_auth 0 2024-03-29T11:25:37.911 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.926+0000 7facd4d70640 10 monclient: _check_auth_tickets 2024-03-29T11:25:37.911 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.926+0000 7facd4d70640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:37.911 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.926+0000 7facd4d70640 1 -- 172.21.15.86:0/3028201641 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7facc0001690 con 0x7facd006ebd0 2024-03-29T11:25:37.911 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.926+0000 7facd4d70640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:37.911 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.926+0000 7facd4d70640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:37.911 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.926+0000 7facd4d70640 10 monclient: dump: 2024-03-29T11:25:37.911 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:37.911 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:37.911 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:37.911 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:37.911 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:37.911 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:37.911 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:37.912 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:37.912 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:37.912 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:37.912 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.926+0000 7facd7fd6640 5 monclient: authenticate success, global_id 4433 2024-03-29T11:25:37.912 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.926+0000 7facd7fd6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:37.912 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.926+0000 7facd4d70640 1 -- 172.21.15.86:0/3028201641 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7facc0001860 con 0x7facd006ebd0 2024-03-29T11:25:37.912 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.926+0000 7facd4d70640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:37.912 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.926+0000 7facd4d70640 1 -- 172.21.15.86:0/3028201641 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7facc0001cb0 con 0x7facd006ebd0 2024-03-29T11:25:37.912 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.926+0000 7facd7fd6640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:37.913 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.926+0000 7facd7fd6640 4 set_mon_vals no callback set 2024-03-29T11:25:37.913 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.927+0000 7facd7fd6640 10 monclient: shutdown 2024-03-29T11:25:37.913 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.927+0000 7facd7fd6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:37.913 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.927+0000 7facd7fd6640 1 -- 172.21.15.86:0/3028201641 >> v1:172.21.15.86:6790/0 conn(0x7facd006ebd0 legacy=0x7facd013b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:37.913 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.927+0000 7facd4d70640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:25:37.913 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.928+0000 7facd7fd6640 1 -- 172.21.15.86:0/3028201641 shutdown_connections 2024-03-29T11:25:37.913 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.928+0000 7facd7fd6640 1 -- 172.21.15.86:0/3028201641 >> 172.21.15.86:0/3028201641 conn(0x7facd00b4200 msgr2=0x7facd00b4610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:37.913 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.928+0000 7facd7fd6640 1 -- 172.21.15.86:0/3028201641 shutdown_connections 2024-03-29T11:25:37.914 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.930+0000 7facd7fd6640 1 -- 172.21.15.86:0/3028201641 wait complete. 2024-03-29T11:25:37.915 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.930+0000 7facd7fd6640 10 monclient: build_initial_monmap 2024-03-29T11:25:37.915 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.931+0000 7facd7fd6640 10 monclient: monmap: 2024-03-29T11:25:37.915 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:25:37.915 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:37.915 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:25:37.915 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:25:37.915 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:37.915 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:37.915 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:25:37.915 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:25:37.915 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:25:37.916 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:37.916 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.931+0000 7facd7fd6640 1 librados: starting msgr at 2024-03-29T11:25:37.917 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.931+0000 7facd7fd6640 1 librados: starting objecter 2024-03-29T11:25:37.917 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.931+0000 7facd7fd6640 1 Processor -- start 2024-03-29T11:25:37.917 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.932+0000 7facd7fd6640 1 -- start start 2024-03-29T11:25:37.917 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.932+0000 7facd7fd6640 1 librados: setting wanted keys 2024-03-29T11:25:37.917 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.932+0000 7facd7fd6640 1 librados: calling monclient init 2024-03-29T11:25:37.917 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.932+0000 7facd7fd6640 10 monclient: init 2024-03-29T11:25:37.917 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.933+0000 7facd7fd6640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:37.918 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.933+0000 7facd7fd6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:37.918 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.933+0000 7facd7fd6640 10 monclient(hunting): picked mon.noname-a con 0x7facd006ebd0 addr v1:172.21.15.86:6789/0 2024-03-29T11:25:37.920 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.936+0000 7facd7fd6640 10 monclient(hunting): picked mon.noname-b con 0x7facd013ed90 addr v1:172.21.15.196:6789/0 2024-03-29T11:25:37.920 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.936+0000 7facd5571640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7facd013ed90 0x7facd00b34e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:45393/0 (socket says 172.21.15.86:45393) 2024-03-29T11:25:37.920 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.936+0000 7facd5571640 1 -- 172.21.15.86:0/1081110530 learned_addr learned my addr 172.21.15.86:0/1081110530 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:25:37.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.939+0000 7facc67fc640 10 client.?.objecter ms_handle_connect 0x7facd006ebd0 2024-03-29T11:25:37.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.939+0000 7facc67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:37.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.939+0000 7facc67fc640 10 client.?.objecter ms_handle_connect 0x7facd013ed90 2024-03-29T11:25:37.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.939+0000 7facc67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:37.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.939+0000 7facd7fd6640 10 monclient(hunting): picked mon.noname-c con 0x7facd013d690 addr v1:172.21.15.86:6790/0 2024-03-29T11:25:37.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.939+0000 7facd7fd6640 1 -- 172.21.15.86:0/1081110530 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd01ca850 con 0x7facd006ebd0 2024-03-29T11:25:37.924 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.939+0000 7facd7fd6640 1 -- 172.21.15.86:0/1081110530 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd01cb3f0 con 0x7facd013ed90 2024-03-29T11:25:37.925 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.939+0000 7facd7fd6640 1 -- 172.21.15.86:0/1081110530 --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7facd01cc120 con 0x7facd013d690 2024-03-29T11:25:37.925 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.939+0000 7facd7fd6640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:37.925 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.939+0000 7facd7fd6640 10 monclient(hunting): authenticate will time out at 1162.257830s 2024-03-29T11:25:37.925 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.940+0000 7facc67fc640 10 client.?.objecter ms_handle_connect 0x7facd013d690 2024-03-29T11:25:37.925 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.940+0000 7facc67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:37.927 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 1 -- 172.21.15.86:0/1081110530 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2210824627 0 0) 0x7facd01cb3f0 con 0x7facd013ed90 2024-03-29T11:25:37.927 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:37.927 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:37.927 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:37.927 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 10 monclient(hunting): my global_id is 4387 2024-03-29T11:25:37.927 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 1 -- 172.21.15.86:0/1081110530 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facb4004eb0 con 0x7facd013ed90 2024-03-29T11:25:37.927 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 1 -- 172.21.15.86:0/1081110530 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1561694505 0 0) 0x7facd01cc120 con 0x7facd013d690 2024-03-29T11:25:37.927 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:25:37.927 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:37.927 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:37.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 10 monclient(hunting): my global_id is 4436 2024-03-29T11:25:37.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 1 -- 172.21.15.86:0/1081110530 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facd01cb3f0 con 0x7facd013d690 2024-03-29T11:25:37.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 1 -- 172.21.15.86:0/1081110530 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1587071892 0 0) 0x7facd01cb3f0 con 0x7facd013d690 2024-03-29T11:25:37.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 1 -- 172.21.15.86:0/1081110530 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7facd01cc120 con 0x7facd013d690 2024-03-29T11:25:37.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 1 -- 172.21.15.86:0/1081110530 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7facc8004230 con 0x7facd013d690 2024-03-29T11:25:37.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:37.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:37.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 10 monclient(hunting): dump: 2024-03-29T11:25:37.928 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:37.928 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:37.928 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:37.928 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:37.928 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:37.928 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:37.928 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:37.928 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:37.929 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:37.929 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:37.929 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:37.929 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 1 -- 172.21.15.86:0/1081110530 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3826991635 0 0) 0x7facb4004eb0 con 0x7facd013ed90 2024-03-29T11:25:37.929 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 1 -- 172.21.15.86:0/1081110530 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7facb40069d0 con 0x7facd013ed90 2024-03-29T11:25:37.929 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 1 -- 172.21.15.86:0/1081110530 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7faccc001870 con 0x7facd013ed90 2024-03-29T11:25:37.929 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:37.929 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:25:37.929 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 10 monclient(hunting): dump: 2024-03-29T11:25:37.929 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:37.929 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:37.929 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:37.929 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:37.929 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:37.930 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:37.930 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:37.930 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:37.930 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:37.930 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:37.930 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 1 -- 172.21.15.86:0/1081110530 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2498753848 0 0) 0x7facd01cc120 con 0x7facd013d690 2024-03-29T11:25:37.930 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 1 -- 172.21.15.86:0/1081110530 >> v1:172.21.15.196:6789/0 conn(0x7facd013ed90 legacy=0x7facd00b34e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:37.930 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 1 -- 172.21.15.86:0/1081110530 >> v1:172.21.15.86:6789/0 conn(0x7facd006ebd0 legacy=0x7facd01cace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:37.930 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:25:37.930 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 1 monclient: found mon.c 2024-03-29T11:25:37.930 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:37.930 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:25:37.930 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 1 -- 172.21.15.86:0/1081110530 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facd00b3bf0 con 0x7facd013d690 2024-03-29T11:25:37.930 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 10 monclient: _finish_auth 0 2024-03-29T11:25:37.931 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:25:37.931 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.942+0000 7facc67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:37.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.943+0000 7facd7fd6640 5 monclient: authenticate success, global_id 4436 2024-03-29T11:25:37.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.943+0000 7facd7fd6640 10 monclient: _renew_subs 2024-03-29T11:25:37.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.943+0000 7facd7fd6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:25:37.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.943+0000 7facd7fd6640 1 -- 172.21.15.86:0/1081110530 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7facd0134160 con 0x7facd013d690 2024-03-29T11:25:37.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.943+0000 7facd7fd6640 10 client.4436.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:37.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.943+0000 7facd7fd6640 10 monclient: _renew_subs 2024-03-29T11:25:37.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.943+0000 7facd7fd6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:25:37.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.943+0000 7facd7fd6640 1 -- 172.21.15.86:0/1081110530 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7facd01cca70 con 0x7facd013d690 2024-03-29T11:25:37.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.943+0000 7facd7fd6640 1 librados: init done 2024-03-29T11:25:37.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.946+0000 7facc67fc640 1 -- 172.21.15.86:0/1081110530 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7facc8004720 con 0x7facd013d690 2024-03-29T11:25:37.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.946+0000 7facc67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:37.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.946+0000 7facc67fc640 1 -- 172.21.15.86:0/1081110530 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7facc8004230 con 0x7facd013d690 2024-03-29T11:25:37.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.946+0000 7facc67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:37.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.946+0000 7facc67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:37.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.946+0000 7facc7fff640 4 set_mon_vals no callback set 2024-03-29T11:25:37.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.946+0000 7facc67fc640 10 monclient: dump: 2024-03-29T11:25:37.934 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:37.934 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:37.934 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:37.934 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:37.934 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:37.934 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:37.935 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:37.935 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:37.935 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:37.935 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:37.935 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.948+0000 7facbbfff640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:37.935 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.948+0000 7facbbfff640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:37.935 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.948+0000 7facbbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:25:37.935 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.948+0000 7facc67fc640 1 -- 172.21.15.86:0/1081110530 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7facc8001250 con 0x7facd013d690 2024-03-29T11:25:37.935 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.949+0000 7facbbfff640 1 -- 172.21.15.86:0/1081110530 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7facb0005080 con 0x7facd013d690 2024-03-29T11:25:37.936 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.949+0000 7facc67fc640 1 -- 172.21.15.86:0/1081110530 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7facc8004050 con 0x7facd013d690 2024-03-29T11:25:37.936 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.949+0000 7facc67fc640 10 client.4436.objecter ms_dispatch 0x7facd0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:37.936 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.949+0000 7facc67fc640 3 client.4436.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:37.936 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.949+0000 7facc67fc640 3 client.4436.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:37.936 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.949+0000 7facc67fc640 20 client.4436.objecter dump_active .. 0 homeless 2024-03-29T11:25:37.936 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.949+0000 7facd5d72640 10 monclient: get_auth_request con 0x7facb406e600 auth_method 0 2024-03-29T11:25:37.936 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.951+0000 7facc67fc640 10 client.4436.objecter ms_handle_connect 0x7facb406e600 2024-03-29T11:25:37.943 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.959+0000 7facc67fc640 1 -- 172.21.15.86:0/1081110530 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7facc80507f0 con 0x7facd013d690 2024-03-29T11:25:37.943 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.959+0000 7facc67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:37.943 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:37.959+0000 7facc67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:38.277 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:38.292+0000 7facbbfff640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:38.277 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:38.292+0000 7facbbfff640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:38.277 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:38.292+0000 7facbbfff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:25:38.277 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:38.292+0000 7facbbfff640 1 -- 172.21.15.86:0/1081110530 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7facb0005330 con 0x7facd013d690 2024-03-29T11:25:38.278 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:38.293+0000 7facc67fc640 1 -- 172.21.15.86:0/1081110530 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7facc8050990 con 0x7facd013d690 2024-03-29T11:25:38.278 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:38.293+0000 7facc67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:38.278 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:38.293+0000 7facc67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:38.280 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:25:38.280 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:38.283 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:38.299+0000 7facbbfff640 10 librados: watch_flush enter 2024-03-29T11:25:38.288 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:38.303+0000 7facbbfff640 10 librados: watch_flush exit 2024-03-29T11:25:38.288 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:38.303+0000 7facbbfff640 20 client.4436.objecter shutdown clearing up homeless session... 2024-03-29T11:25:38.288 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:38.303+0000 7facbbfff640 10 client.4436.objecter successfully canceled tick 2024-03-29T11:25:38.288 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:38.303+0000 7facbbfff640 1 -- 172.21.15.86:0/1081110530 >> v1:172.21.15.196:6816/588110318 conn(0x7facb406e600 legacy=0x7facb4070ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:38.288 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:38.303+0000 7facbbfff640 10 monclient: shutdown 2024-03-29T11:25:38.288 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:38.303+0000 7facbbfff640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:38.288 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:38.303+0000 7facbbfff640 1 -- 172.21.15.86:0/1081110530 >> v1:172.21.15.86:6790/0 conn(0x7facd013d690 legacy=0x7facd0133a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:38.288 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:38.303+0000 7facbbfff640 1 -- 172.21.15.86:0/1081110530 shutdown_connections 2024-03-29T11:25:38.288 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:38.303+0000 7facbbfff640 1 -- 172.21.15.86:0/1081110530 >> 172.21.15.86:0/1081110530 conn(0x7facd00b4200 msgr2=0x7facd013b700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:38.288 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:38.303+0000 7facbbfff640 1 -- 172.21.15.86:0/1081110530 shutdown_connections 2024-03-29T11:25:38.288 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:38.303+0000 7facbbfff640 1 -- 172.21.15.86:0/1081110530 wait complete. 2024-03-29T11:25:38.289 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:38.303+0000 7facbbfff640 1 librados: shutdown 2024-03-29T11:25:38.303 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:45.109 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:45.261 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.274+0000 7f139db17640 10 monclient: get_monmap_and_config 2024-03-29T11:25:45.261 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.274+0000 7f139db17640 10 monclient: build_initial_monmap 2024-03-29T11:25:45.261 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.274+0000 7f139db17640 10 monclient: monmap: 2024-03-29T11:25:45.261 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:25:45.261 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:45.261 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:25:45.275670+0000 2024-03-29T11:25:45.261 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:25:45.275670+0000 2024-03-29T11:25:45.261 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:45.261 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:45.261 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:25:45.262 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:25:45.262 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:25:45.262 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:45.263 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.277+0000 7f139db17640 1 Processor -- start 2024-03-29T11:25:45.263 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.279+0000 7f139db17640 1 -- start start 2024-03-29T11:25:45.264 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.279+0000 7f139db17640 10 monclient: init 2024-03-29T11:25:45.267 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.280+0000 7f139db17640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:45.267 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.280+0000 7f139db17640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:45.267 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.280+0000 7f139db17640 10 monclient(hunting): picked mon.noname-c con 0x7f1390034850 addr v1:172.21.15.196:6789/0 2024-03-29T11:25:45.267 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.280+0000 7f139db17640 10 monclient(hunting): picked mon.noname-a con 0x7f13900c51c0 addr v1:172.21.15.86:6789/0 2024-03-29T11:25:45.268 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.280+0000 7f139db17640 10 monclient(hunting): picked mon.noname-b con 0x7f13900c7cd0 addr v1:172.21.15.86:6790/0 2024-03-29T11:25:45.268 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.280+0000 7f139db17640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1390029190 con 0x7f13900c51c0 2024-03-29T11:25:45.268 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.280+0000 7f139db17640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13900293e0 con 0x7f1390034850 2024-03-29T11:25:45.268 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.280+0000 7f139db17640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13900c8340 con 0x7f13900c7cd0 2024-03-29T11:25:45.268 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.280+0000 7f139db17640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:45.268 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.280+0000 7f139db17640 10 monclient(hunting): authenticate will time out at 1169.598412s 2024-03-29T11:25:45.271 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.286+0000 7f139cb15640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f1390034850 0x7f13900c4ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:46585/0 (socket says 172.21.15.86:46585) 2024-03-29T11:25:45.271 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.286+0000 7f1397fff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f13900c51c0 0x7f1390029800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:42343/0 (socket says 172.21.15.86:42343) 2024-03-29T11:25:45.271 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.286+0000 7f1397fff640 1 -- 172.21.15.86:0/2613246413 learned_addr learned my addr 172.21.15.86:0/2613246413 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:25:45.278 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.287+0000 7f13977fe640 1 -- 172.21.15.86:0/2613246413 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2691850271 0 0) 0x7f1390029190 con 0x7f13900c51c0 2024-03-29T11:25:45.278 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.287+0000 7f13977fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:45.278 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.287+0000 7f13977fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:45.278 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.287+0000 7f13977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:45.278 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.287+0000 7f13977fe640 10 monclient(hunting): my global_id is 4404 2024-03-29T11:25:45.278 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.287+0000 7f13977fe640 1 -- 172.21.15.86:0/2613246413 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1388003520 con 0x7f13900c51c0 2024-03-29T11:25:45.278 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.287+0000 7f13977fe640 1 -- 172.21.15.86:0/2613246413 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4211578902 0 0) 0x7f13900293e0 con 0x7f1390034850 2024-03-29T11:25:45.278 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.287+0000 7f13977fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:45.278 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.287+0000 7f13977fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:45.278 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.287+0000 7f13977fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:45.278 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.287+0000 7f13977fe640 10 monclient(hunting): my global_id is 4390 2024-03-29T11:25:45.279 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.287+0000 7f13977fe640 1 -- 172.21.15.86:0/2613246413 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1388003840 con 0x7f1390034850 2024-03-29T11:25:45.279 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.288+0000 7f13977fe640 1 -- 172.21.15.86:0/2613246413 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1736294199 0 0) 0x7f1388003520 con 0x7f13900c51c0 2024-03-29T11:25:45.279 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.288+0000 7f13977fe640 1 -- 172.21.15.86:0/2613246413 >> v1:172.21.15.86:6790/0 conn(0x7f13900c7cd0 legacy=0x7f1390029f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:45.279 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.288+0000 7f13977fe640 1 -- 172.21.15.86:0/2613246413 >> v1:172.21.15.196:6789/0 conn(0x7f1390034850 legacy=0x7f13900c4ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:45.279 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.288+0000 7f13977fe640 10 monclient: _finish_hunting 0 2024-03-29T11:25:45.279 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.288+0000 7f13977fe640 1 monclient: found mon.noname-a 2024-03-29T11:25:45.279 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.288+0000 7f13977fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:45.279 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.288+0000 7f13977fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.86:6789/0 2024-03-29T11:25:45.279 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.288+0000 7f13977fe640 1 -- 172.21.15.86:0/2613246413 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1390032980 con 0x7f13900c51c0 2024-03-29T11:25:45.279 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.288+0000 7f13977fe640 10 monclient: _finish_auth 0 2024-03-29T11:25:45.280 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.288+0000 7f13977fe640 10 monclient: _check_auth_tickets 2024-03-29T11:25:45.280 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.288+0000 7f13977fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:45.280 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.288+0000 7f13977fe640 1 -- 172.21.15.86:0/2613246413 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f138c001670 con 0x7f13900c51c0 2024-03-29T11:25:45.280 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.288+0000 7f13977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:45.280 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.288+0000 7f13977fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:45.280 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.288+0000 7f13977fe640 10 monclient: dump: 2024-03-29T11:25:45.280 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:45.280 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:45.280 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:45.280 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:45.280 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:45.280 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:45.280 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:45.281 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:45.281 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:45.281 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:45.281 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.288+0000 7f139db17640 5 monclient: authenticate success, global_id 4404 2024-03-29T11:25:45.281 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.288+0000 7f139db17640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:45.281 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.291+0000 7f13977fe640 1 -- 172.21.15.86:0/2613246413 <== mon.0 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f138c001850 con 0x7f13900c51c0 2024-03-29T11:25:45.281 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.291+0000 7f13977fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:45.281 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.291+0000 7f139db17640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:45.281 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.291+0000 7f139db17640 4 set_mon_vals no callback set 2024-03-29T11:25:45.281 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.292+0000 7f13977fe640 1 -- 172.21.15.86:0/2613246413 <== mon.0 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f138c001c60 con 0x7f13900c51c0 2024-03-29T11:25:45.281 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.292+0000 7f13977fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:45.281 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.292+0000 7f13977fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:45.281 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.292+0000 7f13977fe640 10 monclient: dump: 2024-03-29T11:25:45.282 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:45.282 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:45.282 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:45.282 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:45.282 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:45.282 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:45.282 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:45.282 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:45.282 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:45.282 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:45.282 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.292+0000 7f139db17640 10 monclient: shutdown 2024-03-29T11:25:45.282 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.292+0000 7f139db17640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:45.283 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.292+0000 7f139db17640 1 -- 172.21.15.86:0/2613246413 >> v1:172.21.15.86:6789/0 conn(0x7f13900c51c0 legacy=0x7f1390029800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:45.283 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.296+0000 7f139db17640 1 -- 172.21.15.86:0/2613246413 shutdown_connections 2024-03-29T11:25:45.283 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.296+0000 7f139db17640 1 -- 172.21.15.86:0/2613246413 >> 172.21.15.86:0/2613246413 conn(0x7f13900058f0 msgr2=0x7f1390005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:45.283 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.297+0000 7f139db17640 1 -- 172.21.15.86:0/2613246413 shutdown_connections 2024-03-29T11:25:45.284 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.297+0000 7f139db17640 1 -- 172.21.15.86:0/2613246413 wait complete. 2024-03-29T11:25:45.284 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.298+0000 7f139db17640 10 monclient: build_initial_monmap 2024-03-29T11:25:45.284 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.298+0000 7f139db17640 10 monclient: monmap: 2024-03-29T11:25:45.284 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:25:45.284 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:45.284 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:25:45.284 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:25:45.284 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:45.284 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:45.285 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:25:45.285 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:25:45.285 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:25:45.285 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:45.285 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.298+0000 7f139db17640 1 librados: starting msgr at 2024-03-29T11:25:45.286 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.298+0000 7f139db17640 1 librados: starting objecter 2024-03-29T11:25:45.286 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.298+0000 7f139db17640 1 Processor -- start 2024-03-29T11:25:45.286 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.299+0000 7f139db17640 1 -- start start 2024-03-29T11:25:45.286 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.299+0000 7f139db17640 1 librados: setting wanted keys 2024-03-29T11:25:45.286 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.299+0000 7f139db17640 1 librados: calling monclient init 2024-03-29T11:25:45.286 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.299+0000 7f139db17640 10 monclient: init 2024-03-29T11:25:45.286 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.300+0000 7f139db17640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:45.286 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.300+0000 7f139db17640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:25:45.288 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.300+0000 7f139db17640 10 monclient(hunting): picked mon.noname-c con 0x7f1390034850 addr v1:172.21.15.86:6790/0 2024-03-29T11:25:45.288 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.303+0000 7f139db17640 10 monclient(hunting): picked mon.noname-a con 0x7f13900c51c0 addr v1:172.21.15.86:6789/0 2024-03-29T11:25:45.288 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.303+0000 7f139db17640 10 monclient(hunting): picked mon.noname-b con 0x7f13900c7cd0 addr v1:172.21.15.196:6789/0 2024-03-29T11:25:45.288 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.303+0000 7f139db17640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13900b4850 con 0x7f13900c51c0 2024-03-29T11:25:45.288 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.303+0000 7f139db17640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13900b8040 con 0x7f13900c7cd0 2024-03-29T11:25:45.288 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.303+0000 7f139db17640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13900b8320 con 0x7f1390034850 2024-03-29T11:25:45.288 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.303+0000 7f139db17640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:45.288 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.303+0000 7f139db17640 10 monclient(hunting): authenticate will time out at 1169.621179s 2024-03-29T11:25:45.293 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.308+0000 7f139cb15640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f1390034850 0x7f13900c5610 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:33439/0 (socket says 172.21.15.86:33439) 2024-03-29T11:25:45.293 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.308+0000 7f139cb15640 1 -- 172.21.15.86:0/2869570590 learned_addr learned my addr 172.21.15.86:0/2869570590 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:25:45.297 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.308+0000 7f13957fa640 10 client.?.objecter ms_handle_connect 0x7f1390034850 2024-03-29T11:25:45.297 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.309+0000 7f13957fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:45.297 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.309+0000 7f13957fa640 10 client.?.objecter ms_handle_connect 0x7f13900c51c0 2024-03-29T11:25:45.297 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.309+0000 7f13957fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:45.297 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.309+0000 7f13957fa640 1 -- 172.21.15.86:0/2869570590 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4088924924 0 0) 0x7f13900b4850 con 0x7f13900c51c0 2024-03-29T11:25:45.297 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.309+0000 7f13957fa640 10 monclient(hunting): _negotiate 2024-03-29T11:25:45.298 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.309+0000 7f13957fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:45.298 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.309+0000 7f13957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:45.298 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.310+0000 7f13957fa640 10 monclient(hunting): my global_id is 4407 2024-03-29T11:25:45.298 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.310+0000 7f13957fa640 1 -- 172.21.15.86:0/2869570590 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f137c004eb0 con 0x7f13900c51c0 2024-03-29T11:25:45.298 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.310+0000 7f13957fa640 1 -- 172.21.15.86:0/2869570590 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1795745429 0 0) 0x7f13900b8320 con 0x7f1390034850 2024-03-29T11:25:45.298 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.310+0000 7f13957fa640 10 monclient(hunting): _negotiate 2024-03-29T11:25:45.298 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.310+0000 7f13957fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:45.298 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.310+0000 7f13957fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:45.298 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.310+0000 7f13957fa640 10 monclient(hunting): my global_id is 4439 2024-03-29T11:25:45.298 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.310+0000 7f13957fa640 1 -- 172.21.15.86:0/2869570590 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13900b4850 con 0x7f1390034850 2024-03-29T11:25:45.298 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.310+0000 7f13957fa640 1 -- 172.21.15.86:0/2869570590 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2595565054 0 0) 0x7f137c004eb0 con 0x7f13900c51c0 2024-03-29T11:25:45.298 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.310+0000 7f13957fa640 1 -- 172.21.15.86:0/2869570590 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f13900b8320 con 0x7f13900c51c0 2024-03-29T11:25:45.298 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.310+0000 7f13957fa640 1 -- 172.21.15.86:0/2869570590 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f138c001b60 con 0x7f13900c51c0 2024-03-29T11:25:45.299 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.310+0000 7f13957fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:45.299 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.310+0000 7f13957fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:25:45.299 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.310+0000 7f13957fa640 10 monclient(hunting): dump: 2024-03-29T11:25:45.299 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:45.299 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:45.299 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:45.299 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:45.299 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:45.299 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:45.299 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:45.299 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:45.299 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:45.299 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:45.299 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.310+0000 7f13957fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:45.299 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.310+0000 7f13957fa640 1 -- 172.21.15.86:0/2869570590 <== mon.0 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3093795185 0 0) 0x7f13900b8320 con 0x7f13900c51c0 2024-03-29T11:25:45.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.310+0000 7f13957fa640 1 -- 172.21.15.86:0/2869570590 >> v1:172.21.15.86:6790/0 conn(0x7f1390034850 legacy=0x7f13900c5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:45.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.310+0000 7f13957fa640 1 -- 172.21.15.86:0/2869570590 >> v1:172.21.15.196:6789/0 conn(0x7f13900c7cd0 legacy=0x7f13900b4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:45.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.310+0000 7f13957fa640 10 monclient: _finish_hunting 0 2024-03-29T11:25:45.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.310+0000 7f13957fa640 1 monclient: found mon.a 2024-03-29T11:25:45.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.310+0000 7f13957fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:45.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.310+0000 7f13957fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:25:45.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.310+0000 7f13957fa640 1 -- 172.21.15.86:0/2869570590 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13900b8600 con 0x7f13900c51c0 2024-03-29T11:25:45.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.310+0000 7f13957fa640 10 monclient: _finish_auth 0 2024-03-29T11:25:45.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.310+0000 7f13957fa640 10 monclient: _check_auth_tickets 2024-03-29T11:25:45.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.310+0000 7f13957fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:45.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.310+0000 7f139db17640 5 monclient: authenticate success, global_id 4407 2024-03-29T11:25:45.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.310+0000 7f139db17640 10 monclient: _renew_subs 2024-03-29T11:25:45.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.311+0000 7f139db17640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:25:45.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.311+0000 7f139db17640 1 -- 172.21.15.86:0/2869570590 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13900b71e0 con 0x7f13900c51c0 2024-03-29T11:25:45.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.311+0000 7f139db17640 10 client.4407.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:45.300 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.311+0000 7f139db17640 10 monclient: _renew_subs 2024-03-29T11:25:45.301 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.311+0000 7f139db17640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:25:45.301 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.311+0000 7f139db17640 1 -- 172.21.15.86:0/2869570590 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f13900c5f00 con 0x7f13900c51c0 2024-03-29T11:25:45.301 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.311+0000 7f139db17640 1 librados: init done 2024-03-29T11:25:45.301 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.311+0000 7f13957fa640 1 -- 172.21.15.86:0/2869570590 <== mon.0 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f138c0011f0 con 0x7f13900c51c0 2024-03-29T11:25:45.301 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.311+0000 7f13957fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:45.301 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.311+0000 7f13957fa640 1 -- 172.21.15.86:0/2869570590 <== mon.0 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f138c004cb0 con 0x7f13900c51c0 2024-03-29T11:25:45.301 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.311+0000 7f13957fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:45.301 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.311+0000 7f13957fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:25:45.301 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.311+0000 7f13957fa640 10 monclient: dump: 2024-03-29T11:25:45.301 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:45.301 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:45.301 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:45.301 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:45.301 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:45.301 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:45.301 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:45.302 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:45.302 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:45.302 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:45.303 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.313+0000 7f1396ffd640 4 set_mon_vals no callback set 2024-03-29T11:25:45.303 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.314+0000 7f13957fa640 1 -- 172.21.15.86:0/2869570590 <== mon.0 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f138c01ac80 con 0x7f13900c51c0 2024-03-29T11:25:45.303 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.314+0000 7f139db17640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:45.303 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.315+0000 7f139db17640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:45.303 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.315+0000 7f139db17640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:25:45.303 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.315+0000 7f139db17640 1 -- 172.21.15.86:0/2869570590 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1378005080 con 0x7f13900c51c0 2024-03-29T11:25:45.304 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.319+0000 7f13957fa640 1 -- 172.21.15.86:0/2869570590 <== mon.0 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f138c084c20 con 0x7f13900c51c0 2024-03-29T11:25:45.304 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.319+0000 7f13957fa640 10 client.4407.objecter ms_dispatch 0x7f1398000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:45.304 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.319+0000 7f139cb15640 10 monclient: get_auth_request con 0x7f137c06de00 auth_method 0 2024-03-29T11:25:45.304 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.319+0000 7f13957fa640 3 client.4407.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:45.304 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.319+0000 7f13957fa640 3 client.4407.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:45.304 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.319+0000 7f13957fa640 20 client.4407.objecter dump_active .. 0 homeless 2024-03-29T11:25:45.304 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.320+0000 7f13957fa640 10 client.4407.objecter ms_handle_connect 0x7f137c06de00 2024-03-29T11:25:45.313 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.328+0000 7f13957fa640 1 -- 172.21.15.86:0/2869570590 <== mon.0 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f138c050500 con 0x7f13900c51c0 2024-03-29T11:25:45.313 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.328+0000 7f13957fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:45.313 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.328+0000 7f13957fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:45.644 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.659+0000 7f139db17640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:45.644 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.659+0000 7f139db17640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:45.644 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.659+0000 7f139db17640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:25:45.644 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.659+0000 7f139db17640 1 -- 172.21.15.86:0/2869570590 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1378005330 con 0x7f13900c51c0 2024-03-29T11:25:45.648 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.664+0000 7f13957fa640 1 -- 172.21.15.86:0/2869570590 <== mon.0 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f138c052430 con 0x7f13900c51c0 2024-03-29T11:25:45.648 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.664+0000 7f13957fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:45.648 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.664+0000 7f13957fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:45.650 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:25:45.650 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:45.652 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.667+0000 7f139db17640 10 librados: watch_flush enter 2024-03-29T11:25:45.652 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.667+0000 7f139db17640 10 librados: watch_flush exit 2024-03-29T11:25:45.652 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.667+0000 7f139db17640 20 client.4407.objecter shutdown clearing up homeless session... 2024-03-29T11:25:45.652 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.667+0000 7f139db17640 10 client.4407.objecter successfully canceled tick 2024-03-29T11:25:45.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.667+0000 7f139db17640 1 -- 172.21.15.86:0/2869570590 >> v1:172.21.15.196:6816/588110318 conn(0x7f137c06de00 legacy=0x7f137c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:45.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.667+0000 7f139db17640 10 monclient: shutdown 2024-03-29T11:25:45.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.667+0000 7f139db17640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:45.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.667+0000 7f139db17640 1 -- 172.21.15.86:0/2869570590 >> v1:172.21.15.86:6789/0 conn(0x7f13900c51c0 legacy=0x7f13900b3a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:45.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.668+0000 7f139db17640 1 -- 172.21.15.86:0/2869570590 shutdown_connections 2024-03-29T11:25:45.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.668+0000 7f139db17640 1 -- 172.21.15.86:0/2869570590 >> 172.21.15.86:0/2869570590 conn(0x7f13900058f0 msgr2=0x7f13900c4880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:45.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.668+0000 7f139db17640 1 -- 172.21.15.86:0/2869570590 shutdown_connections 2024-03-29T11:25:45.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.668+0000 7f139db17640 1 -- 172.21.15.86:0/2869570590 wait complete. 2024-03-29T11:25:45.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:45.668+0000 7f139db17640 1 librados: shutdown 2024-03-29T11:25:45.670 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:52.477 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:25:52.630 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.642+0000 7f204943e640 10 monclient: get_monmap_and_config 2024-03-29T11:25:52.630 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.642+0000 7f204943e640 10 monclient: build_initial_monmap 2024-03-29T11:25:52.630 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.642+0000 7f204943e640 10 monclient: monmap: 2024-03-29T11:25:52.630 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:25:52.630 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:52.630 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:25:52.643024+0000 2024-03-29T11:25:52.630 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:25:52.643024+0000 2024-03-29T11:25:52.630 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:52.631 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:52.631 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:25:52.631 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:25:52.631 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:25:52.631 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:52.631 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.643+0000 7f204943e640 1 Processor -- start 2024-03-29T11:25:52.631 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.643+0000 7f204943e640 1 -- start start 2024-03-29T11:25:52.631 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.643+0000 7f204943e640 10 monclient: init 2024-03-29T11:25:52.631 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.643+0000 7f204943e640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:52.631 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.643+0000 7f204943e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:25:52.631 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.644+0000 7f204943e640 10 monclient(hunting): picked mon.noname-c con 0x7f20440b0860 addr v1:172.21.15.196:6789/0 2024-03-29T11:25:52.631 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.644+0000 7f204943e640 10 monclient(hunting): picked mon.noname-a con 0x7f204413ef30 addr v1:172.21.15.86:6789/0 2024-03-29T11:25:52.631 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.644+0000 7f204943e640 10 monclient(hunting): picked mon.noname-b con 0x7f204406d9d0 addr v1:172.21.15.86:6790/0 2024-03-29T11:25:52.631 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.644+0000 7f204943e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20440b0c40 con 0x7f204413ef30 2024-03-29T11:25:52.631 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.644+0000 7f204943e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20440aed10 con 0x7f20440b0860 2024-03-29T11:25:52.632 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.644+0000 7f204943e640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20440ab1b0 con 0x7f204406d9d0 2024-03-29T11:25:52.632 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.644+0000 7f204943e640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:52.632 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.644+0000 7f204943e640 10 monclient(hunting): authenticate will time out at 1176.962094s 2024-03-29T11:25:52.633 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.647+0000 7f20437fe640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f204406d9d0 0x7f204414a300 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:55552/0 (socket says 172.21.15.86:55552) 2024-03-29T11:25:52.633 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.647+0000 7f20437fe640 1 -- 172.21.15.86:0/4198645661 learned_addr learned my addr 172.21.15.86:0/4198645661 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:25:52.633 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.647+0000 7f2041ffb640 1 -- 172.21.15.86:0/4198645661 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1529609442 0 0) 0x7f20440ab1b0 con 0x7f204406d9d0 2024-03-29T11:25:52.633 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.647+0000 7f2041ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:25:52.634 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.647+0000 7f2041ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:52.634 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.647+0000 7f2041ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:52.634 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.647+0000 7f2041ffb640 10 monclient(hunting): my global_id is 4442 2024-03-29T11:25:52.634 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.647+0000 7f2041ffb640 1 -- 172.21.15.86:0/4198645661 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f202c003520 con 0x7f204406d9d0 2024-03-29T11:25:52.634 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.647+0000 7f2041ffb640 1 -- 172.21.15.86:0/4198645661 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1396430116 0 0) 0x7f202c003520 con 0x7f204406d9d0 2024-03-29T11:25:52.634 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.647+0000 7f2041ffb640 1 -- 172.21.15.86:0/4198645661 >> v1:172.21.15.196:6789/0 conn(0x7f20440b0860 legacy=0x7f204413e820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:25:52.634 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.647+0000 7f2041ffb640 1 -- 172.21.15.86:0/4198645661 >> v1:172.21.15.86:6789/0 conn(0x7f204413ef30 legacy=0x7f20440af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:52.634 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.647+0000 7f2041ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:25:52.634 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.647+0000 7f2041ffb640 1 monclient: found mon.noname-b 2024-03-29T11:25:52.634 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.647+0000 7f2041ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:52.634 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.647+0000 7f2041ffb640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:25:52.634 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.647+0000 7f2041ffb640 1 -- 172.21.15.86:0/4198645661 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20440ab490 con 0x7f204406d9d0 2024-03-29T11:25:52.634 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.647+0000 7f2041ffb640 10 monclient: _finish_auth 0 2024-03-29T11:25:52.635 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.647+0000 7f2041ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:25:52.635 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.647+0000 7f2041ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:52.635 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.647+0000 7f2041ffb640 1 -- 172.21.15.86:0/4198645661 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f2038001690 con 0x7f204406d9d0 2024-03-29T11:25:52.635 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.647+0000 7f2041ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:52.635 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.647+0000 7f2041ffb640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:25:52.635 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.647+0000 7f2041ffb640 10 monclient: dump: 2024-03-29T11:25:52.635 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:52.635 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:52.635 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:52.635 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:52.635 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:52.635 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:52.635 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:52.635 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:52.635 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:52.636 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:52.636 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.648+0000 7f204943e640 5 monclient: authenticate success, global_id 4442 2024-03-29T11:25:52.636 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.648+0000 7f204943e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:25:52.636 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.648+0000 7f2041ffb640 1 -- 172.21.15.86:0/4198645661 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2038001860 con 0x7f204406d9d0 2024-03-29T11:25:52.636 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.648+0000 7f2041ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:52.637 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.649+0000 7f2041ffb640 1 -- 172.21.15.86:0/4198645661 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f2038001cb0 con 0x7f204406d9d0 2024-03-29T11:25:52.637 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.649+0000 7f2041ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:52.637 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.649+0000 7f2041ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:52.637 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.649+0000 7f2041ffb640 10 monclient: dump: 2024-03-29T11:25:52.637 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:52.637 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:52.637 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:52.637 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:52.637 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:52.637 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:52.637 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:52.637 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:52.637 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:52.638 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:52.638 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.650+0000 7f204943e640 10 monclient: get_monmap_and_config success 2024-03-29T11:25:52.638 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.650+0000 7f204943e640 4 set_mon_vals no callback set 2024-03-29T11:25:52.638 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.651+0000 7f204943e640 10 monclient: shutdown 2024-03-29T11:25:52.638 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.651+0000 7f204943e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:52.638 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.651+0000 7f204943e640 1 -- 172.21.15.86:0/4198645661 >> v1:172.21.15.86:6790/0 conn(0x7f204406d9d0 legacy=0x7f204414a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:52.638 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.652+0000 7f204943e640 1 -- 172.21.15.86:0/4198645661 shutdown_connections 2024-03-29T11:25:52.638 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.652+0000 7f204943e640 1 -- 172.21.15.86:0/4198645661 >> 172.21.15.86:0/4198645661 conn(0x7f204406b660 msgr2=0x7f204406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:52.638 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.652+0000 7f204943e640 1 -- 172.21.15.86:0/4198645661 shutdown_connections 2024-03-29T11:25:52.638 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.653+0000 7f204943e640 1 -- 172.21.15.86:0/4198645661 wait complete. 2024-03-29T11:25:52.640 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.654+0000 7f204943e640 10 monclient: build_initial_monmap 2024-03-29T11:25:52.640 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.654+0000 7f204943e640 10 monclient: monmap: 2024-03-29T11:25:52.640 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:25:52.640 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:25:52.640 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:25:52.640 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:25:52.640 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:25:52.640 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:52.640 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:25:52.640 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:25:52.640 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:25:52.641 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:52.641 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.654+0000 7f204943e640 1 librados: starting msgr at 2024-03-29T11:25:52.641 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.654+0000 7f204943e640 1 librados: starting objecter 2024-03-29T11:25:52.641 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.655+0000 7f204943e640 1 Processor -- start 2024-03-29T11:25:52.644 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.655+0000 7f204943e640 1 -- start start 2024-03-29T11:25:52.644 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.655+0000 7f204943e640 1 librados: setting wanted keys 2024-03-29T11:25:52.644 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.655+0000 7f204943e640 1 librados: calling monclient init 2024-03-29T11:25:52.644 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.655+0000 7f204943e640 10 monclient: init 2024-03-29T11:25:52.645 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.656+0000 7f204943e640 10 monclient: _reopen_session rank -1 2024-03-29T11:25:52.645 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.656+0000 7f204943e640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:25:52.645 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.656+0000 7f204943e640 10 monclient(hunting): picked mon.noname-a con 0x7f204406d9d0 addr v1:172.21.15.86:6789/0 2024-03-29T11:25:52.645 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.656+0000 7f204943e640 10 monclient(hunting): picked mon.noname-b con 0x7f20440b0860 addr v1:172.21.15.196:6789/0 2024-03-29T11:25:52.645 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.656+0000 7f204943e640 10 monclient(hunting): picked mon.noname-c con 0x7f204413ef30 addr v1:172.21.15.86:6790/0 2024-03-29T11:25:52.645 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.656+0000 7f204943e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2044071b80 con 0x7f204406d9d0 2024-03-29T11:25:52.645 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.656+0000 7f204943e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2044070a70 con 0x7f20440b0860 2024-03-29T11:25:52.645 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.656+0000 7f204943e640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f204406f700 con 0x7f204413ef30 2024-03-29T11:25:52.645 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.656+0000 7f204943e640 10 monclient(hunting): _renew_subs 2024-03-29T11:25:52.645 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.656+0000 7f204943e640 10 monclient(hunting): authenticate will time out at 1176.974383s 2024-03-29T11:25:52.645 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.657+0000 7f20437fe640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f204413ef30 0x7f20441e5ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:59410/0 (socket says 172.21.15.86:59410) 2024-03-29T11:25:52.645 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.657+0000 7f20437fe640 1 -- 172.21.15.86:0/324323110 learned_addr learned my addr 172.21.15.86:0/324323110 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:25:52.645 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.657+0000 7f202b7fe640 10 client.?.objecter ms_handle_connect 0x7f204413ef30 2024-03-29T11:25:52.645 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.657+0000 7f202b7fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:25:52.646 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.657+0000 7f202b7fe640 1 -- 172.21.15.86:0/324323110 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3621668510 0 0) 0x7f204406f700 con 0x7f204413ef30 2024-03-29T11:25:52.646 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.657+0000 7f202b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:25:52.646 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.657+0000 7f202b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:25:52.646 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.657+0000 7f202b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:25:52.646 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.657+0000 7f202b7fe640 10 monclient(hunting): my global_id is 4445 2024-03-29T11:25:52.646 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.658+0000 7f202b7fe640 1 -- 172.21.15.86:0/324323110 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2030004eb0 con 0x7f204413ef30 2024-03-29T11:25:52.646 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.658+0000 7f202b7fe640 1 -- 172.21.15.86:0/324323110 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4118863856 0 0) 0x7f2030004eb0 con 0x7f204413ef30 2024-03-29T11:25:52.646 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.658+0000 7f202b7fe640 1 -- 172.21.15.86:0/324323110 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f204406f700 con 0x7f204413ef30 2024-03-29T11:25:52.646 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.658+0000 7f202b7fe640 1 -- 172.21.15.86:0/324323110 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f2038001b70 con 0x7f204413ef30 2024-03-29T11:25:52.646 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.658+0000 7f202b7fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:52.646 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.658+0000 7f202b7fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:25:52.646 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.658+0000 7f202b7fe640 10 monclient(hunting): dump: 2024-03-29T11:25:52.646 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:52.646 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:52.646 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:52.647 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:52.647 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:52.647 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:52.647 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:52.647 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:52.647 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:52.647 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:52.647 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.659+0000 7f202b7fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:25:52.647 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.659+0000 7f202b7fe640 1 -- 172.21.15.86:0/324323110 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2090066221 0 0) 0x7f204406f700 con 0x7f204413ef30 2024-03-29T11:25:52.647 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.659+0000 7f202b7fe640 1 -- 172.21.15.86:0/324323110 >> v1:172.21.15.196:6789/0 conn(0x7f20440b0860 legacy=0x7f204406eff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:52.647 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.660+0000 7f202b7fe640 1 -- 172.21.15.86:0/324323110 >> v1:172.21.15.86:6789/0 conn(0x7f204406d9d0 legacy=0x7f2044071470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:25:52.650 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.660+0000 7f202b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:25:52.651 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.661+0000 7f202b7fe640 1 monclient: found mon.c 2024-03-29T11:25:52.651 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.661+0000 7f202b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:25:52.651 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.661+0000 7f202b7fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:25:52.651 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.661+0000 7f202b7fe640 1 -- 172.21.15.86:0/324323110 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f204406f890 con 0x7f204413ef30 2024-03-29T11:25:52.651 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.661+0000 7f202b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:25:52.651 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.661+0000 7f202b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:25:52.651 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.661+0000 7f202b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:25:52.651 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.661+0000 7f204943e640 5 monclient: authenticate success, global_id 4445 2024-03-29T11:25:52.651 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.661+0000 7f204943e640 10 monclient: _renew_subs 2024-03-29T11:25:52.651 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.661+0000 7f204943e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:25:52.651 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.661+0000 7f204943e640 1 -- 172.21.15.86:0/324323110 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f204406e610 con 0x7f204413ef30 2024-03-29T11:25:52.651 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.661+0000 7f204943e640 10 client.4445.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:25:52.651 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.661+0000 7f204943e640 10 monclient: _renew_subs 2024-03-29T11:25:52.651 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.661+0000 7f204943e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:25:52.651 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.661+0000 7f204943e640 1 -- 172.21.15.86:0/324323110 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f204406e9b0 con 0x7f204413ef30 2024-03-29T11:25:52.652 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.661+0000 7f204943e640 1 librados: init done 2024-03-29T11:25:52.652 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.662+0000 7f202b7fe640 1 -- 172.21.15.86:0/324323110 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2038004bf0 con 0x7f204413ef30 2024-03-29T11:25:52.652 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.662+0000 7f202b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:25:52.652 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.662+0000 7f2040ff9640 4 set_mon_vals no callback set 2024-03-29T11:25:52.652 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.662+0000 7f202b7fe640 1 -- 172.21.15.86:0/324323110 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f2038004f00 con 0x7f204413ef30 2024-03-29T11:25:52.652 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.662+0000 7f202b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:25:52.652 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.662+0000 7f202b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:25:52.652 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.662+0000 7f202b7fe640 10 monclient: dump: 2024-03-29T11:25:52.652 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:25:52.652 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:25:52.652 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:52.652 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:25:52.652 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:25:52.652 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:25:52.652 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:25:52.653 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:25:52.653 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:25:52.653 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:25:52.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.663+0000 7f202b7fe640 1 -- 172.21.15.86:0/324323110 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f203801aed0 con 0x7f204413ef30 2024-03-29T11:25:52.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.664+0000 7f2042ffd640 10 monclient: get_auth_request con 0x7f203006dc60 auth_method 0 2024-03-29T11:25:52.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.664+0000 7f202b7fe640 1 -- 172.21.15.86:0/324323110 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f2038084bc0 con 0x7f204413ef30 2024-03-29T11:25:52.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.664+0000 7f202b7fe640 10 client.4445.objecter ms_dispatch 0x7f2044000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:25:52.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.664+0000 7f202b7fe640 3 client.4445.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:25:52.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.664+0000 7f202b7fe640 3 client.4445.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:25:52.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.664+0000 7f204943e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:52.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.664+0000 7f204943e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:52.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.665+0000 7f204943e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:25:52.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.665+0000 7f204943e640 1 -- 172.21.15.86:0/324323110 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2024005080 con 0x7f204413ef30 2024-03-29T11:25:52.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.665+0000 7f202b7fe640 20 client.4445.objecter dump_active .. 0 homeless 2024-03-29T11:25:52.653 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.665+0000 7f202b7fe640 10 client.4445.objecter ms_handle_connect 0x7f203006dc60 2024-03-29T11:25:52.659 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.674+0000 7f202b7fe640 1 -- 172.21.15.86:0/324323110 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f20380504a0 con 0x7f204413ef30 2024-03-29T11:25:52.659 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.674+0000 7f202b7fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:25:52.659 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:52.674+0000 7f202b7fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:25:53.028 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:53.043+0000 7f204943e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:53.028 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:53.043+0000 7f204943e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:53.028 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:53.043+0000 7f204943e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:25:53.028 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:53.043+0000 7f204943e640 1 -- 172.21.15.86:0/324323110 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2024005330 con 0x7f204413ef30 2024-03-29T11:25:53.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:53.044+0000 7f202b7fe640 1 -- 172.21.15.86:0/324323110 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f20380523d0 con 0x7f204413ef30 2024-03-29T11:25:53.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:53.044+0000 7f202b7fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:25:53.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:53.044+0000 7f202b7fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:25:53.030 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:25:53.030 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:25:53.034 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:53.049+0000 7f204943e640 10 librados: watch_flush enter 2024-03-29T11:25:53.034 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:53.049+0000 7f204943e640 10 librados: watch_flush exit 2024-03-29T11:25:53.034 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:53.049+0000 7f204943e640 20 client.4445.objecter shutdown clearing up homeless session... 2024-03-29T11:25:53.034 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:53.049+0000 7f204943e640 10 client.4445.objecter successfully canceled tick 2024-03-29T11:25:53.036 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:53.052+0000 7f204943e640 1 -- 172.21.15.86:0/324323110 >> v1:172.21.15.196:6816/588110318 conn(0x7f203006dc60 legacy=0x7f2030070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:53.036 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:53.052+0000 7f204943e640 10 monclient: shutdown 2024-03-29T11:25:53.037 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:53.052+0000 7f204943e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:25:53.037 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:53.052+0000 7f204943e640 1 -- 172.21.15.86:0/324323110 >> v1:172.21.15.86:6790/0 conn(0x7f204413ef30 legacy=0x7f20441e5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:25:53.037 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:53.052+0000 7f204943e640 1 -- 172.21.15.86:0/324323110 shutdown_connections 2024-03-29T11:25:53.037 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:53.052+0000 7f204943e640 1 -- 172.21.15.86:0/324323110 >> 172.21.15.86:0/324323110 conn(0x7f204406b660 msgr2=0x7f204406ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:25:53.038 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:53.053+0000 7f204943e640 1 -- 172.21.15.86:0/324323110 shutdown_connections 2024-03-29T11:25:53.038 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:53.053+0000 7f204943e640 1 -- 172.21.15.86:0/324323110 wait complete. 2024-03-29T11:25:53.038 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:25:53.053+0000 7f204943e640 1 librados: shutdown 2024-03-29T11:25:53.059 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:25:59.867 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:00.017 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.031+0000 7fec3e082640 10 monclient: get_monmap_and_config 2024-03-29T11:26:00.017 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.031+0000 7fec3e082640 10 monclient: build_initial_monmap 2024-03-29T11:26:00.018 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.031+0000 7fec3e082640 10 monclient: monmap: 2024-03-29T11:26:00.018 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:26:00.018 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:00.018 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:26:00.032866+0000 2024-03-29T11:26:00.018 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:26:00.032866+0000 2024-03-29T11:26:00.018 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:00.018 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:00.018 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:26:00.018 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:26:00.018 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:26:00.018 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:00.020 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.033+0000 7fec3e082640 1 Processor -- start 2024-03-29T11:26:00.020 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.033+0000 7fec3e082640 1 -- start start 2024-03-29T11:26:00.020 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.033+0000 7fec3e082640 10 monclient: init 2024-03-29T11:26:00.021 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.034+0000 7fec3e082640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:00.021 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.034+0000 7fec3e082640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:00.021 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.034+0000 7fec3e082640 10 monclient(hunting): picked mon.noname-b con 0x7fec380a9750 addr v1:172.21.15.86:6790/0 2024-03-29T11:26:00.021 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.034+0000 7fec3e082640 10 monclient(hunting): picked mon.noname-c con 0x7fec380ae220 addr v1:172.21.15.196:6789/0 2024-03-29T11:26:00.021 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.035+0000 7fec3e082640 10 monclient(hunting): picked mon.noname-a con 0x7fec3813b9d0 addr v1:172.21.15.86:6789/0 2024-03-29T11:26:00.021 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.035+0000 7fec3e082640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec38142490 con 0x7fec3813b9d0 2024-03-29T11:26:00.021 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.035+0000 7fec3e082640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec380ac130 con 0x7fec380ae220 2024-03-29T11:26:00.021 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.035+0000 7fec3e082640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec380ac410 con 0x7fec380a9750 2024-03-29T11:26:00.021 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.035+0000 7fec3e082640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:00.021 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.035+0000 7fec3e082640 10 monclient(hunting): authenticate will time out at 1184.353306s 2024-03-29T11:26:00.024 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.039+0000 7fec3d080640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7fec380a9750 0x7fec38141d80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:52210/0 (socket says 172.21.15.86:52210) 2024-03-29T11:26:00.024 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.039+0000 7fec3d080640 1 -- 172.21.15.86:0/3377766001 learned_addr learned my addr 172.21.15.86:0/3377766001 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:26:00.026 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.040+0000 7fec2ffff640 1 -- 172.21.15.86:0/3377766001 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2271374928 0 0) 0x7fec380ac410 con 0x7fec380a9750 2024-03-29T11:26:00.026 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.040+0000 7fec2ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:00.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.040+0000 7fec2ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:00.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.040+0000 7fec2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:00.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.040+0000 7fec2ffff640 10 monclient(hunting): my global_id is 4448 2024-03-29T11:26:00.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.040+0000 7fec2ffff640 1 -- 172.21.15.86:0/3377766001 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec24003520 con 0x7fec380a9750 2024-03-29T11:26:00.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.041+0000 7fec2ffff640 1 -- 172.21.15.86:0/3377766001 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 780036674 0 0) 0x7fec380ac130 con 0x7fec380ae220 2024-03-29T11:26:00.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.041+0000 7fec2ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:00.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.041+0000 7fec2ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:00.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.041+0000 7fec2ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:00.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.041+0000 7fec2ffff640 10 monclient(hunting): my global_id is 4393 2024-03-29T11:26:00.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.041+0000 7fec2ffff640 1 -- 172.21.15.86:0/3377766001 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec380ac410 con 0x7fec380ae220 2024-03-29T11:26:00.028 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.041+0000 7fec2ffff640 1 -- 172.21.15.86:0/3377766001 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4110286652 0 0) 0x7fec24003520 con 0x7fec380a9750 2024-03-29T11:26:00.028 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.041+0000 7fec2ffff640 1 -- 172.21.15.86:0/3377766001 >> v1:172.21.15.196:6789/0 conn(0x7fec380ae220 legacy=0x7fec3813b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:00.028 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.041+0000 7fec2ffff640 1 -- 172.21.15.86:0/3377766001 >> v1:172.21.15.86:6789/0 conn(0x7fec3813b9d0 legacy=0x7fec38149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:00.028 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.041+0000 7fec2ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:00.028 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.041+0000 7fec2ffff640 1 monclient: found mon.noname-b 2024-03-29T11:26:00.028 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.041+0000 7fec2ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:00.028 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.041+0000 7fec2ffff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:26:00.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.041+0000 7fec2ffff640 1 -- 172.21.15.86:0/3377766001 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec380afa40 con 0x7fec380a9750 2024-03-29T11:26:00.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.041+0000 7fec2ffff640 10 monclient: _finish_auth 0 2024-03-29T11:26:00.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.041+0000 7fec2ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:00.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.041+0000 7fec2ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:00.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.041+0000 7fec2ffff640 1 -- 172.21.15.86:0/3377766001 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fec34001610 con 0x7fec380a9750 2024-03-29T11:26:00.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.041+0000 7fec2ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:00.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.041+0000 7fec2ffff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:00.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.041+0000 7fec2ffff640 10 monclient: dump: 2024-03-29T11:26:00.029 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:00.030 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:00.030 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:00.030 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:00.030 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:00.030 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:00.030 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:00.030 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:00.030 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:00.030 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:00.031 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.041+0000 7fec3e082640 5 monclient: authenticate success, global_id 4448 2024-03-29T11:26:00.031 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.041+0000 7fec3e082640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:00.031 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.041+0000 7fec2ffff640 1 -- 172.21.15.86:0/3377766001 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec34001770 con 0x7fec380a9750 2024-03-29T11:26:00.031 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.041+0000 7fec2ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:00.031 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.041+0000 7fec2ffff640 1 -- 172.21.15.86:0/3377766001 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fec34001bc0 con 0x7fec380a9750 2024-03-29T11:26:00.031 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.041+0000 7fec3e082640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:00.031 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.041+0000 7fec3e082640 4 set_mon_vals no callback set 2024-03-29T11:26:00.031 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.041+0000 7fec2ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:00.031 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.041+0000 7fec2ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:00.031 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.041+0000 7fec2ffff640 10 monclient: dump: 2024-03-29T11:26:00.031 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:00.031 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:00.031 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:00.031 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:00.031 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:00.032 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:00.032 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:00.032 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:00.032 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:00.032 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:00.033 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.042+0000 7fec3e082640 10 monclient: shutdown 2024-03-29T11:26:00.033 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.042+0000 7fec3e082640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:00.033 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.042+0000 7fec3e082640 1 -- 172.21.15.86:0/3377766001 >> v1:172.21.15.86:6790/0 conn(0x7fec380a9750 legacy=0x7fec38141d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:00.033 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.042+0000 7fec3e082640 1 -- 172.21.15.86:0/3377766001 shutdown_connections 2024-03-29T11:26:00.033 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.042+0000 7fec3e082640 1 -- 172.21.15.86:0/3377766001 >> 172.21.15.86:0/3377766001 conn(0x7fec380673e0 msgr2=0x7fec380b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:00.034 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.042+0000 7fec3e082640 1 -- 172.21.15.86:0/3377766001 shutdown_connections 2024-03-29T11:26:00.034 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.043+0000 7fec3e082640 1 -- 172.21.15.86:0/3377766001 wait complete. 2024-03-29T11:26:00.034 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.043+0000 7fec3e082640 10 monclient: build_initial_monmap 2024-03-29T11:26:00.034 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.043+0000 7fec3e082640 10 monclient: monmap: 2024-03-29T11:26:00.034 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:26:00.034 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:00.034 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:26:00.034 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:26:00.035 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:00.035 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:00.035 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:26:00.035 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:26:00.035 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:26:00.035 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:00.035 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.044+0000 7fec3e082640 1 librados: starting msgr at 2024-03-29T11:26:00.035 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.044+0000 7fec3e082640 1 librados: starting objecter 2024-03-29T11:26:00.035 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.044+0000 7fec3e082640 1 Processor -- start 2024-03-29T11:26:00.035 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.044+0000 7fec3e082640 1 -- start start 2024-03-29T11:26:00.035 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.044+0000 7fec3e082640 1 librados: setting wanted keys 2024-03-29T11:26:00.035 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.044+0000 7fec3e082640 1 librados: calling monclient init 2024-03-29T11:26:00.035 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.044+0000 7fec3e082640 10 monclient: init 2024-03-29T11:26:00.035 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.045+0000 7fec3e082640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:00.036 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.045+0000 7fec3e082640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:00.036 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.045+0000 7fec3e082640 10 monclient(hunting): picked mon.noname-c con 0x7fec380a9750 addr v1:172.21.15.86:6790/0 2024-03-29T11:26:00.036 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.045+0000 7fec3e082640 10 monclient(hunting): picked mon.noname-a con 0x7fec380ae220 addr v1:172.21.15.86:6789/0 2024-03-29T11:26:00.036 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.046+0000 7fec3e082640 10 monclient(hunting): picked mon.noname-b con 0x7fec3813b9d0 addr v1:172.21.15.196:6789/0 2024-03-29T11:26:00.036 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.046+0000 7fec3e082640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec3806f9c0 con 0x7fec380ae220 2024-03-29T11:26:00.036 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.046+0000 7fec3e082640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec3806e730 con 0x7fec3813b9d0 2024-03-29T11:26:00.036 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.046+0000 7fec3e082640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec38134310 con 0x7fec380a9750 2024-03-29T11:26:00.036 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.046+0000 7fec3e082640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:00.036 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.046+0000 7fec3e082640 10 monclient(hunting): authenticate will time out at 1184.364263s 2024-03-29T11:26:00.041 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.049+0000 7fec3d080640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7fec380a9750 0x7fec38133c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:52654/0 (socket says 172.21.15.86:52654) 2024-03-29T11:26:00.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.049+0000 7fec3d881640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fec3813b9d0 0x7fec381d6930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:46547/0 (socket says 172.21.15.86:46547) 2024-03-29T11:26:00.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.049+0000 7fec3d881640 1 -- 172.21.15.86:0/2961849621 learned_addr learned my addr 172.21.15.86:0/2961849621 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:26:00.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.049+0000 7fec2dffb640 10 client.?.objecter ms_handle_connect 0x7fec3813b9d0 2024-03-29T11:26:00.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.049+0000 7fec2dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:00.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.049+0000 7fec2dffb640 10 client.?.objecter ms_handle_connect 0x7fec380a9750 2024-03-29T11:26:00.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.049+0000 7fec2dffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:00.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.049+0000 7fec2dffb640 1 -- 172.21.15.86:0/2961849621 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3728453286 0 0) 0x7fec3806e730 con 0x7fec3813b9d0 2024-03-29T11:26:00.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.049+0000 7fec2dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:00.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.049+0000 7fec2dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:00.043 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.049+0000 7fec2dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:00.043 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.049+0000 7fec2dffb640 10 monclient(hunting): my global_id is 4396 2024-03-29T11:26:00.043 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.049+0000 7fec2dffb640 1 -- 172.21.15.86:0/2961849621 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec1c004eb0 con 0x7fec3813b9d0 2024-03-29T11:26:00.043 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.050+0000 7fec2dffb640 1 -- 172.21.15.86:0/2961849621 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2360738648 0 0) 0x7fec38134310 con 0x7fec380a9750 2024-03-29T11:26:00.043 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.050+0000 7fec2dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:00.043 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.050+0000 7fec2dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:00.043 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.050+0000 7fec2dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:00.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.050+0000 7fec2dffb640 10 monclient(hunting): my global_id is 4451 2024-03-29T11:26:00.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.050+0000 7fec2dffb640 1 -- 172.21.15.86:0/2961849621 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec3806e730 con 0x7fec380a9750 2024-03-29T11:26:00.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.050+0000 7fec2dffb640 1 -- 172.21.15.86:0/2961849621 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 107682221 0 0) 0x7fec1c004eb0 con 0x7fec3813b9d0 2024-03-29T11:26:00.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.050+0000 7fec2dffb640 1 -- 172.21.15.86:0/2961849621 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec38134310 con 0x7fec3813b9d0 2024-03-29T11:26:00.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.050+0000 7fec2dffb640 1 -- 172.21.15.86:0/2961849621 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fec30001ad0 con 0x7fec3813b9d0 2024-03-29T11:26:00.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.050+0000 7fec2dffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:00.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.050+0000 7fec2dffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:00.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.050+0000 7fec2dffb640 10 monclient(hunting): dump: 2024-03-29T11:26:00.044 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:00.044 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:00.044 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:00.045 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:00.045 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:00.045 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:00.045 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:00.045 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:00.045 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:00.045 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:00.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.050+0000 7fec2dffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:00.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.050+0000 7fec2dffb640 1 -- 172.21.15.86:0/2961849621 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1538632624 0 0) 0x7fec38134310 con 0x7fec3813b9d0 2024-03-29T11:26:00.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.050+0000 7fec2dffb640 1 -- 172.21.15.86:0/2961849621 >> v1:172.21.15.86:6790/0 conn(0x7fec380a9750 legacy=0x7fec38133c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:00.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.050+0000 7fec2dffb640 1 -- 172.21.15.86:0/2961849621 >> v1:172.21.15.86:6789/0 conn(0x7fec380ae220 legacy=0x7fec381d6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:00.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.050+0000 7fec2dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:26:00.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.050+0000 7fec2dffb640 1 monclient: found mon.b 2024-03-29T11:26:00.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.051+0000 7fec2dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:00.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.051+0000 7fec2dffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:26:00.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.051+0000 7fec2dffb640 1 -- 172.21.15.86:0/2961849621 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec3806d8c0 con 0x7fec3813b9d0 2024-03-29T11:26:00.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.051+0000 7fec2dffb640 10 monclient: _finish_auth 0 2024-03-29T11:26:00.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.051+0000 7fec2dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:26:00.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.051+0000 7fec2dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:00.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.051+0000 7fec3e082640 5 monclient: authenticate success, global_id 4396 2024-03-29T11:26:00.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.051+0000 7fec3e082640 10 monclient: _renew_subs 2024-03-29T11:26:00.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.051+0000 7fec3e082640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:26:00.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.051+0000 7fec3e082640 1 -- 172.21.15.86:0/2961849621 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec380b3450 con 0x7fec3813b9d0 2024-03-29T11:26:00.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.051+0000 7fec2dffb640 1 -- 172.21.15.86:0/2961849621 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec30001c30 con 0x7fec3813b9d0 2024-03-29T11:26:00.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.051+0000 7fec3e082640 10 client.4396.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:00.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.051+0000 7fec3e082640 10 monclient: _renew_subs 2024-03-29T11:26:00.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.051+0000 7fec3e082640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:26:00.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.051+0000 7fec3e082640 1 -- 172.21.15.86:0/2961849621 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fec3806e2a0 con 0x7fec3813b9d0 2024-03-29T11:26:00.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.051+0000 7fec3e082640 1 librados: init done 2024-03-29T11:26:00.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.051+0000 7fec2dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:00.047 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.051+0000 7fec2dffb640 1 -- 172.21.15.86:0/2961849621 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fec30004d80 con 0x7fec3813b9d0 2024-03-29T11:26:00.047 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.051+0000 7fec2dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:00.047 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.051+0000 7fec2dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:00.047 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.051+0000 7fec2dffb640 10 monclient: dump: 2024-03-29T11:26:00.047 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:00.047 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:00.047 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:00.047 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:00.047 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:00.047 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:00.047 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:00.047 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:00.047 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:00.047 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:00.047 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.053+0000 7fec2dffb640 1 -- 172.21.15.86:0/2961849621 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7fec3001ad50 con 0x7fec3813b9d0 2024-03-29T11:26:00.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.054+0000 7fec2dffb640 1 -- 172.21.15.86:0/2961849621 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7fec30084d00 con 0x7fec3813b9d0 2024-03-29T11:26:00.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.054+0000 7fec2dffb640 10 client.4396.objecter ms_dispatch 0x7fec38000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:00.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.054+0000 7fec3d080640 10 monclient: get_auth_request con 0x7fec1c06dd30 auth_method 0 2024-03-29T11:26:00.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.054+0000 7fec2dffb640 3 client.4396.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:00.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.054+0000 7fec2dffb640 3 client.4396.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:00.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.054+0000 7fec2dffb640 20 client.4396.objecter dump_active .. 0 homeless 2024-03-29T11:26:00.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.054+0000 7fec2dffb640 10 client.4396.objecter ms_handle_connect 0x7fec1c06dd30 2024-03-29T11:26:00.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.054+0000 7fec3e082640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:00.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.054+0000 7fec2effd640 4 set_mon_vals no callback set 2024-03-29T11:26:00.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.055+0000 7fec3e082640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:00.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.055+0000 7fec3e082640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:26:00.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.056+0000 7fec3e082640 1 -- 172.21.15.86:0/2961849621 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec3806e8e0 con 0x7fec3813b9d0 2024-03-29T11:26:00.049 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.064+0000 7fec2dffb640 1 -- 172.21.15.86:0/2961849621 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fec3806e8e0 con 0x7fec3813b9d0 2024-03-29T11:26:00.049 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.064+0000 7fec2dffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:00.049 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.064+0000 7fec2dffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:00.371 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.385+0000 7fec3e082640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:00.371 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.385+0000 7fec3e082640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:00.371 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.385+0000 7fec3e082640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:26:00.371 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.385+0000 7fec3e082640 1 -- 172.21.15.86:0/2961849621 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fec3806e8e0 con 0x7fec3813b9d0 2024-03-29T11:26:00.373 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:26:00.373 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:00.374 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.388+0000 7fec2dffb640 1 -- 172.21.15.86:0/2961849621 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7fec3806e8e0 con 0x7fec3813b9d0 2024-03-29T11:26:00.374 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.388+0000 7fec2dffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:00.374 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.388+0000 7fec2dffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:00.378 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.392+0000 7fec3e082640 10 librados: watch_flush enter 2024-03-29T11:26:00.378 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.392+0000 7fec3e082640 10 librados: watch_flush exit 2024-03-29T11:26:00.378 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.392+0000 7fec3e082640 20 client.4396.objecter shutdown clearing up homeless session... 2024-03-29T11:26:00.378 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.392+0000 7fec3e082640 10 client.4396.objecter successfully canceled tick 2024-03-29T11:26:00.378 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.393+0000 7fec3e082640 1 -- 172.21.15.86:0/2961849621 >> v1:172.21.15.196:6816/588110318 conn(0x7fec1c06dd30 legacy=0x7fec1c0701f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:00.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.393+0000 7fec3e082640 10 monclient: shutdown 2024-03-29T11:26:00.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.393+0000 7fec3e082640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:00.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.393+0000 7fec3e082640 1 -- 172.21.15.86:0/2961849621 >> v1:172.21.15.196:6789/0 conn(0x7fec3813b9d0 legacy=0x7fec381d6930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:00.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.393+0000 7fec3e082640 1 -- 172.21.15.86:0/2961849621 shutdown_connections 2024-03-29T11:26:00.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.393+0000 7fec3e082640 1 -- 172.21.15.86:0/2961849621 >> 172.21.15.86:0/2961849621 conn(0x7fec380673e0 msgr2=0x7fec38069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:00.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.393+0000 7fec3e082640 1 -- 172.21.15.86:0/2961849621 shutdown_connections 2024-03-29T11:26:00.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.393+0000 7fec3e082640 1 -- 172.21.15.86:0/2961849621 wait complete. 2024-03-29T11:26:00.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:00.393+0000 7fec3e082640 1 librados: shutdown 2024-03-29T11:26:00.396 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:07.204 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:07.368 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.382+0000 7fec1cc86640 10 monclient: get_monmap_and_config 2024-03-29T11:26:07.368 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.382+0000 7fec1cc86640 10 monclient: build_initial_monmap 2024-03-29T11:26:07.368 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.382+0000 7fec1cc86640 10 monclient: monmap: 2024-03-29T11:26:07.368 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:26:07.369 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:07.369 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:26:07.383656+0000 2024-03-29T11:26:07.369 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:26:07.383656+0000 2024-03-29T11:26:07.369 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:07.369 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:07.369 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:26:07.369 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:26:07.369 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:26:07.369 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:07.370 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.383+0000 7fec1cc86640 1 Processor -- start 2024-03-29T11:26:07.370 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.383+0000 7fec1cc86640 1 -- start start 2024-03-29T11:26:07.371 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.383+0000 7fec1cc86640 10 monclient: init 2024-03-29T11:26:07.371 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.384+0000 7fec1cc86640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:07.371 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.384+0000 7fec1cc86640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:26:07.371 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.384+0000 7fec1cc86640 10 monclient(hunting): picked mon.noname-a con 0x7fec180b0770 addr v1:172.21.15.86:6789/0 2024-03-29T11:26:07.371 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.384+0000 7fec1cc86640 10 monclient(hunting): picked mon.noname-c con 0x7fec1814a700 addr v1:172.21.15.196:6789/0 2024-03-29T11:26:07.371 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.384+0000 7fec1cc86640 10 monclient(hunting): picked mon.noname-b con 0x7fec180ac5a0 addr v1:172.21.15.86:6790/0 2024-03-29T11:26:07.371 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.384+0000 7fec1cc86640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec180aba70 con 0x7fec180b0770 2024-03-29T11:26:07.371 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.384+0000 7fec1cc86640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec180aecb0 con 0x7fec1814a700 2024-03-29T11:26:07.371 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.384+0000 7fec1cc86640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec180af150 con 0x7fec180ac5a0 2024-03-29T11:26:07.371 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.384+0000 7fec1cc86640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:07.371 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.384+0000 7fec1cc86640 10 monclient(hunting): authenticate will time out at 1191.702878s 2024-03-29T11:26:07.371 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.385+0000 7fec16d9d640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7fec180ac5a0 0x7fec18151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:52666/0 (socket says 172.21.15.86:52666) 2024-03-29T11:26:07.371 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.385+0000 7fec16d9d640 1 -- 172.21.15.86:0/3673509837 learned_addr learned my addr 172.21.15.86:0/3673509837 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:26:07.372 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.386+0000 7fec1559a640 1 -- 172.21.15.86:0/3673509837 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3307806019 0 0) 0x7fec180af150 con 0x7fec180ac5a0 2024-03-29T11:26:07.372 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.386+0000 7fec1559a640 10 monclient(hunting): _negotiate 2024-03-29T11:26:07.372 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.386+0000 7fec1559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:07.372 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.386+0000 7fec1559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:07.373 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.386+0000 7fec1559a640 10 monclient(hunting): my global_id is 4454 2024-03-29T11:26:07.373 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.386+0000 7fec1559a640 1 -- 172.21.15.86:0/3673509837 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec00003520 con 0x7fec180ac5a0 2024-03-29T11:26:07.373 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.386+0000 7fec1559a640 1 -- 172.21.15.86:0/3673509837 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1837330713 0 0) 0x7fec180aba70 con 0x7fec180b0770 2024-03-29T11:26:07.373 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.386+0000 7fec1559a640 10 monclient(hunting): _negotiate 2024-03-29T11:26:07.373 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.386+0000 7fec1559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:07.373 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.386+0000 7fec1559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:07.373 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.386+0000 7fec1559a640 10 monclient(hunting): my global_id is 4413 2024-03-29T11:26:07.373 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.386+0000 7fec1559a640 1 -- 172.21.15.86:0/3673509837 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec180af150 con 0x7fec180b0770 2024-03-29T11:26:07.373 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.386+0000 7fec1559a640 1 -- 172.21.15.86:0/3673509837 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3235627811 0 0) 0x7fec180aecb0 con 0x7fec1814a700 2024-03-29T11:26:07.373 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.386+0000 7fec1559a640 10 monclient(hunting): _negotiate 2024-03-29T11:26:07.373 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.386+0000 7fec1559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:07.373 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.386+0000 7fec1559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:07.373 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.386+0000 7fec1559a640 10 monclient(hunting): my global_id is 4399 2024-03-29T11:26:07.373 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.386+0000 7fec1559a640 1 -- 172.21.15.86:0/3673509837 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec180aba70 con 0x7fec1814a700 2024-03-29T11:26:07.374 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.387+0000 7fec1559a640 1 -- 172.21.15.86:0/3673509837 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 75715005 0 0) 0x7fec180aba70 con 0x7fec1814a700 2024-03-29T11:26:07.374 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.387+0000 7fec1559a640 1 -- 172.21.15.86:0/3673509837 >> v1:172.21.15.86:6790/0 conn(0x7fec180ac5a0 legacy=0x7fec18151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:07.374 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.387+0000 7fec1559a640 1 -- 172.21.15.86:0/3673509837 >> v1:172.21.15.86:6789/0 conn(0x7fec180b0770 legacy=0x7fec18149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:07.374 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.387+0000 7fec1559a640 10 monclient: _finish_hunting 0 2024-03-29T11:26:07.374 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.387+0000 7fec1559a640 1 monclient: found mon.noname-c 2024-03-29T11:26:07.374 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.387+0000 7fec1559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:07.374 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.387+0000 7fec1559a640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T11:26:07.374 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.387+0000 7fec1559a640 1 -- 172.21.15.86:0/3673509837 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec180ac9d0 con 0x7fec1814a700 2024-03-29T11:26:07.375 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.387+0000 7fec1559a640 10 monclient: _finish_auth 0 2024-03-29T11:26:07.375 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.387+0000 7fec1559a640 10 monclient: _check_auth_tickets 2024-03-29T11:26:07.375 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.387+0000 7fec1559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:07.375 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.387+0000 7fec1cc86640 5 monclient: authenticate success, global_id 4399 2024-03-29T11:26:07.375 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.387+0000 7fec1cc86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:07.375 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.387+0000 7fec1559a640 1 -- 172.21.15.86:0/3673509837 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fec0c0015b0 con 0x7fec1814a700 2024-03-29T11:26:07.375 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.387+0000 7fec1559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:07.376 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.387+0000 7fec1559a640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:07.376 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.387+0000 7fec1559a640 10 monclient: dump: 2024-03-29T11:26:07.376 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:07.376 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:07.376 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:07.376 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:07.376 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:07.376 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:07.376 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:07.376 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:07.376 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:07.376 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:07.377 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.387+0000 7fec1cc86640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:07.377 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.387+0000 7fec1559a640 1 -- 172.21.15.86:0/3673509837 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec0c001790 con 0x7fec1814a700 2024-03-29T11:26:07.377 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.387+0000 7fec1559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:07.377 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.387+0000 7fec1cc86640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:07.377 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.387+0000 7fec1cc86640 4 set_mon_vals no callback set 2024-03-29T11:26:07.382 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.387+0000 7fec1559a640 1 -- 172.21.15.86:0/3673509837 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fec0c0015b0 con 0x7fec1814a700 2024-03-29T11:26:07.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.387+0000 7fec1559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:07.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.387+0000 7fec1559a640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:07.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.387+0000 7fec1559a640 10 monclient: dump: 2024-03-29T11:26:07.383 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:07.383 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:07.383 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:07.384 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:07.384 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:07.384 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:07.384 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:07.384 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:07.384 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:07.384 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:07.385 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.388+0000 7fec1cc86640 10 monclient: shutdown 2024-03-29T11:26:07.385 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.388+0000 7fec1cc86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:07.385 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.388+0000 7fec1cc86640 1 -- 172.21.15.86:0/3673509837 >> v1:172.21.15.196:6789/0 conn(0x7fec1814a700 legacy=0x7fec180abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:07.385 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.388+0000 7fec1cc86640 1 -- 172.21.15.86:0/3673509837 shutdown_connections 2024-03-29T11:26:07.385 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.388+0000 7fec1cc86640 1 -- 172.21.15.86:0/3673509837 >> 172.21.15.86:0/3673509837 conn(0x7fec1806b660 msgr2=0x7fec1806ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:07.385 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.388+0000 7fec1cc86640 1 -- 172.21.15.86:0/3673509837 shutdown_connections 2024-03-29T11:26:07.385 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.388+0000 7fec1cc86640 1 -- 172.21.15.86:0/3673509837 wait complete. 2024-03-29T11:26:07.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.390+0000 7fec1cc86640 10 monclient: build_initial_monmap 2024-03-29T11:26:07.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.390+0000 7fec1cc86640 10 monclient: monmap: 2024-03-29T11:26:07.386 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:26:07.386 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:07.386 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:26:07.386 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:26:07.386 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:07.386 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:07.387 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:26:07.387 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:26:07.387 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:26:07.387 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:07.387 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.390+0000 7fec1cc86640 1 librados: starting msgr at 2024-03-29T11:26:07.387 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.390+0000 7fec1cc86640 1 librados: starting objecter 2024-03-29T11:26:07.388 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.390+0000 7fec1cc86640 1 Processor -- start 2024-03-29T11:26:07.388 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.390+0000 7fec1cc86640 1 -- start start 2024-03-29T11:26:07.388 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.390+0000 7fec1cc86640 1 librados: setting wanted keys 2024-03-29T11:26:07.388 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.390+0000 7fec1cc86640 1 librados: calling monclient init 2024-03-29T11:26:07.388 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.390+0000 7fec1cc86640 10 monclient: init 2024-03-29T11:26:07.388 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.391+0000 7fec1cc86640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:07.388 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.391+0000 7fec1cc86640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:07.389 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.391+0000 7fec1cc86640 10 monclient(hunting): picked mon.noname-b con 0x7fec180ac5a0 addr v1:172.21.15.196:6789/0 2024-03-29T11:26:07.389 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.392+0000 7fec1cc86640 10 monclient(hunting): picked mon.noname-a con 0x7fec180b0770 addr v1:172.21.15.86:6789/0 2024-03-29T11:26:07.389 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.392+0000 7fec1cc86640 10 monclient(hunting): picked mon.noname-c con 0x7fec1814a700 addr v1:172.21.15.86:6790/0 2024-03-29T11:26:07.389 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.392+0000 7fec1659c640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fec180ac5a0 0x7fec181d2bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:34694/0 (socket says 172.21.15.86:34694) 2024-03-29T11:26:07.389 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.392+0000 7fec1659c640 1 -- 172.21.15.86:0/3519573576 learned_addr learned my addr 172.21.15.86:0/3519573576 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:26:07.389 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.392+0000 7fec1cc86640 1 -- 172.21.15.86:0/3519573576 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec18124680 con 0x7fec180b0770 2024-03-29T11:26:07.390 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.392+0000 7fec1cc86640 1 -- 172.21.15.86:0/3519573576 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec181d32c0 con 0x7fec180ac5a0 2024-03-29T11:26:07.390 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.392+0000 7fec1cc86640 1 -- 172.21.15.86:0/3519573576 --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec181dc5f0 con 0x7fec1814a700 2024-03-29T11:26:07.390 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.392+0000 7fec1cc86640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:07.390 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.392+0000 7fec1cc86640 10 monclient(hunting): authenticate will time out at 1191.710377s 2024-03-29T11:26:07.390 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.392+0000 7fec06ffd640 10 client.?.objecter ms_handle_connect 0x7fec180ac5a0 2024-03-29T11:26:07.390 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.392+0000 7fec06ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:07.390 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.392+0000 7fec06ffd640 10 client.?.objecter ms_handle_connect 0x7fec1814a700 2024-03-29T11:26:07.390 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.392+0000 7fec06ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:07.390 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.392+0000 7fec06ffd640 1 -- 172.21.15.86:0/3519573576 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2853724603 0 0) 0x7fec181d32c0 con 0x7fec180ac5a0 2024-03-29T11:26:07.390 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.392+0000 7fec06ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:07.390 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.392+0000 7fec06ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:07.390 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.392+0000 7fec06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:07.391 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.392+0000 7fec06ffd640 10 monclient(hunting): my global_id is 4402 2024-03-29T11:26:07.394 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.392+0000 7fec06ffd640 1 -- 172.21.15.86:0/3519573576 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febf8004eb0 con 0x7fec180ac5a0 2024-03-29T11:26:07.394 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.393+0000 7fec06ffd640 1 -- 172.21.15.86:0/3519573576 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3412812389 0 0) 0x7fec181dc5f0 con 0x7fec1814a700 2024-03-29T11:26:07.394 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.393+0000 7fec06ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:07.394 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.393+0000 7fec06ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:07.394 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.393+0000 7fec06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:07.394 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.393+0000 7fec06ffd640 10 monclient(hunting): my global_id is 4457 2024-03-29T11:26:07.394 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.393+0000 7fec06ffd640 1 -- 172.21.15.86:0/3519573576 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec181d32c0 con 0x7fec1814a700 2024-03-29T11:26:07.394 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.393+0000 7fec06ffd640 10 client.?.objecter ms_handle_connect 0x7fec180b0770 2024-03-29T11:26:07.395 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.393+0000 7fec06ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:07.395 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.393+0000 7fec06ffd640 1 -- 172.21.15.86:0/3519573576 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4061542435 0 0) 0x7fec18124680 con 0x7fec180b0770 2024-03-29T11:26:07.395 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.393+0000 7fec06ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:07.395 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.393+0000 7fec06ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:07.395 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.393+0000 7fec06ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:07.395 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.393+0000 7fec06ffd640 10 monclient(hunting): my global_id is 4416 2024-03-29T11:26:07.395 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.393+0000 7fec06ffd640 1 -- 172.21.15.86:0/3519573576 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fec181dc5f0 con 0x7fec180b0770 2024-03-29T11:26:07.395 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.393+0000 7fec06ffd640 1 -- 172.21.15.86:0/3519573576 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1451236346 0 0) 0x7febf8004eb0 con 0x7fec180ac5a0 2024-03-29T11:26:07.395 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.393+0000 7fec06ffd640 1 -- 172.21.15.86:0/3519573576 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec18124680 con 0x7fec180ac5a0 2024-03-29T11:26:07.395 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.393+0000 7fec06ffd640 1 -- 172.21.15.86:0/3519573576 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3356993307 0 0) 0x7fec181d32c0 con 0x7fec1814a700 2024-03-29T11:26:07.395 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.393+0000 7fec06ffd640 1 -- 172.21.15.86:0/3519573576 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7febf8004eb0 con 0x7fec1814a700 2024-03-29T11:26:07.395 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.393+0000 7fec06ffd640 1 -- 172.21.15.86:0/3519573576 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fec080019b0 con 0x7fec180ac5a0 2024-03-29T11:26:07.395 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.393+0000 7fec06ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:07.395 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.394+0000 7fec06ffd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:07.395 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.394+0000 7fec06ffd640 10 monclient(hunting): dump: 2024-03-29T11:26:07.396 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:07.396 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:07.396 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:07.396 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:07.396 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:07.396 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:07.396 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:07.396 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:07.396 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:07.396 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:07.396 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.394+0000 7fec06ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:07.396 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.394+0000 7fec06ffd640 1 -- 172.21.15.86:0/3519573576 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fec10001c50 con 0x7fec1814a700 2024-03-29T11:26:07.396 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.394+0000 7fec06ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:07.396 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.394+0000 7fec06ffd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:07.397 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.394+0000 7fec06ffd640 10 monclient(hunting): dump: 2024-03-29T11:26:07.397 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:07.397 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:07.397 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:07.397 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:07.397 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:07.397 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:07.397 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:07.397 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:07.397 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:07.397 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:07.397 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.394+0000 7fec06ffd640 1 -- 172.21.15.86:0/3519573576 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3654180990 0 0) 0x7fec181dc5f0 con 0x7fec180b0770 2024-03-29T11:26:07.397 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.394+0000 7fec06ffd640 1 -- 172.21.15.86:0/3519573576 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fec181d32c0 con 0x7fec180b0770 2024-03-29T11:26:07.397 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.394+0000 7fec06ffd640 1 -- 172.21.15.86:0/3519573576 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fec0c0015b0 con 0x7fec180b0770 2024-03-29T11:26:07.397 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.394+0000 7fec06ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:07.398 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.394+0000 7fec06ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:07.398 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.394+0000 7fec06ffd640 10 monclient(hunting): dump: 2024-03-29T11:26:07.398 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:07.398 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:07.398 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:07.398 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:07.398 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:07.398 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:07.398 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:07.398 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:07.398 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:07.398 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:07.398 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.394+0000 7fec06ffd640 1 -- 172.21.15.86:0/3519573576 <== mon.0 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2093755037 0 0) 0x7fec181d32c0 con 0x7fec180b0770 2024-03-29T11:26:07.399 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.394+0000 7fec06ffd640 1 -- 172.21.15.86:0/3519573576 >> v1:172.21.15.86:6790/0 conn(0x7fec1814a700 legacy=0x7fec18144530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:07.399 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.394+0000 7fec06ffd640 1 -- 172.21.15.86:0/3519573576 >> v1:172.21.15.196:6789/0 conn(0x7fec180ac5a0 legacy=0x7fec181d2bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:07.399 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.394+0000 7fec06ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:07.399 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.394+0000 7fec06ffd640 1 monclient: found mon.a 2024-03-29T11:26:07.399 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.394+0000 7fec06ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:07.399 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.394+0000 7fec06ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:26:07.399 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.394+0000 7fec06ffd640 1 -- 172.21.15.86:0/3519573576 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec181dc780 con 0x7fec180b0770 2024-03-29T11:26:07.399 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.394+0000 7fec06ffd640 10 monclient: _finish_auth 0 2024-03-29T11:26:07.399 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.394+0000 7fec06ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:07.399 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.394+0000 7fec06ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:07.399 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.395+0000 7fec1cc86640 5 monclient: authenticate success, global_id 4416 2024-03-29T11:26:07.399 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.395+0000 7fec1cc86640 10 monclient: _renew_subs 2024-03-29T11:26:07.399 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.395+0000 7fec1cc86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:26:07.399 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.395+0000 7fec1cc86640 1 -- 172.21.15.86:0/3519573576 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec181d24d0 con 0x7fec180b0770 2024-03-29T11:26:07.399 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.395+0000 7fec1cc86640 10 client.4416.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:07.400 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.395+0000 7fec1cc86640 10 monclient: _renew_subs 2024-03-29T11:26:07.400 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.395+0000 7fec1cc86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:26:07.400 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.395+0000 7fec1cc86640 1 -- 172.21.15.86:0/3519573576 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fec18144e20 con 0x7fec180b0770 2024-03-29T11:26:07.400 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.395+0000 7fec1cc86640 1 librados: init done 2024-03-29T11:26:07.400 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.396+0000 7fec06ffd640 1 -- 172.21.15.86:0/3519573576 <== mon.0 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fec0c001d00 con 0x7fec180b0770 2024-03-29T11:26:07.400 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.396+0000 7fec06ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:07.400 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.396+0000 7fec06ffd640 1 -- 172.21.15.86:0/3519573576 <== mon.0 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fec0c003a40 con 0x7fec180b0770 2024-03-29T11:26:07.400 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.396+0000 7fec06ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:07.400 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.396+0000 7fec06ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:07.400 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.396+0000 7fec14d99640 4 set_mon_vals no callback set 2024-03-29T11:26:07.401 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.396+0000 7fec06ffd640 10 monclient: dump: 2024-03-29T11:26:07.401 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:07.401 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:07.401 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:07.401 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:07.401 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:07.401 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:07.401 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:07.401 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:07.401 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:07.401 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:07.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.398+0000 7fec06ffd640 1 -- 172.21.15.86:0/3519573576 <== mon.0 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7fec0c019a10 con 0x7fec180b0770 2024-03-29T11:26:07.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.399+0000 7fec06ffd640 1 -- 172.21.15.86:0/3519573576 <== mon.0 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7fec0c083800 con 0x7fec180b0770 2024-03-29T11:26:07.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.399+0000 7fec06ffd640 10 client.4416.objecter ms_dispatch 0x7fec18000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:07.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.399+0000 7fec06ffd640 3 client.4416.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:07.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.399+0000 7fec06ffd640 3 client.4416.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:07.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.399+0000 7fec06ffd640 20 client.4416.objecter dump_active .. 0 homeless 2024-03-29T11:26:07.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.400+0000 7fec1659c640 10 monclient: get_auth_request con 0x7febf806e0d0 auth_method 0 2024-03-29T11:26:07.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.400+0000 7fec1cc86640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:07.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.400+0000 7fec1cc86640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:07.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.400+0000 7fec1cc86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:26:07.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.400+0000 7fec1cc86640 1 -- 172.21.15.86:0/3519573576 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febfc005080 con 0x7fec180b0770 2024-03-29T11:26:07.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.400+0000 7fec06ffd640 10 client.4416.objecter ms_handle_connect 0x7febf806e0d0 2024-03-29T11:26:07.404 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.411+0000 7fec06ffd640 1 -- 172.21.15.86:0/3519573576 <== mon.0 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fec0c04f0e0 con 0x7fec180b0770 2024-03-29T11:26:07.404 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.411+0000 7fec06ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:07.404 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.411+0000 7fec06ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:07.818 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.832+0000 7fec1cc86640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:07.818 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.832+0000 7fec1cc86640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:07.818 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.832+0000 7fec1cc86640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:26:07.818 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.832+0000 7fec1cc86640 1 -- 172.21.15.86:0/3519573576 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febfc005330 con 0x7fec180b0770 2024-03-29T11:26:07.819 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.834+0000 7fec06ffd640 1 -- 172.21.15.86:0/3519573576 <== mon.0 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7fec0c051010 con 0x7fec180b0770 2024-03-29T11:26:07.819 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.834+0000 7fec06ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:07.819 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.834+0000 7fec06ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:07.820 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:26:07.820 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:07.824 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.837+0000 7fec1cc86640 10 librados: watch_flush enter 2024-03-29T11:26:07.824 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.837+0000 7fec1cc86640 10 librados: watch_flush exit 2024-03-29T11:26:07.824 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.837+0000 7fec1cc86640 20 client.4416.objecter shutdown clearing up homeless session... 2024-03-29T11:26:07.824 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.837+0000 7fec1cc86640 10 client.4416.objecter successfully canceled tick 2024-03-29T11:26:07.824 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.838+0000 7fec1cc86640 1 -- 172.21.15.86:0/3519573576 >> v1:172.21.15.196:6816/588110318 conn(0x7febf806e0d0 legacy=0x7febf8070590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:07.824 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.838+0000 7fec1cc86640 10 monclient: shutdown 2024-03-29T11:26:07.824 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.838+0000 7fec1cc86640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:07.824 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.838+0000 7fec1cc86640 1 -- 172.21.15.86:0/3519573576 >> v1:172.21.15.86:6789/0 conn(0x7fec180b0770 legacy=0x7fec181dbee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:07.824 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.838+0000 7fec1cc86640 1 -- 172.21.15.86:0/3519573576 shutdown_connections 2024-03-29T11:26:07.824 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.838+0000 7fec1cc86640 1 -- 172.21.15.86:0/3519573576 >> 172.21.15.86:0/3519573576 conn(0x7fec1806b660 msgr2=0x7fec180b3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:07.824 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.838+0000 7fec1cc86640 1 -- 172.21.15.86:0/3519573576 shutdown_connections 2024-03-29T11:26:07.824 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.838+0000 7fec1cc86640 1 -- 172.21.15.86:0/3519573576 wait complete. 2024-03-29T11:26:07.824 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:07.838+0000 7fec1cc86640 1 librados: shutdown 2024-03-29T11:26:07.841 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:14.649 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:14.778 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.792+0000 7fefbccc3640 10 monclient: get_monmap_and_config 2024-03-29T11:26:14.778 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.792+0000 7fefbccc3640 10 monclient: build_initial_monmap 2024-03-29T11:26:14.778 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.792+0000 7fefbccc3640 10 monclient: monmap: 2024-03-29T11:26:14.778 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:26:14.778 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:14.778 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:26:14.793284+0000 2024-03-29T11:26:14.778 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:26:14.793284+0000 2024-03-29T11:26:14.778 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:14.778 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:14.778 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:26:14.778 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:26:14.778 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:26:14.779 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:14.779 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.792+0000 7fefbccc3640 1 Processor -- start 2024-03-29T11:26:14.780 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.793+0000 7fefbccc3640 1 -- start start 2024-03-29T11:26:14.780 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.793+0000 7fefbccc3640 10 monclient: init 2024-03-29T11:26:14.780 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.794+0000 7fefbccc3640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:14.780 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.794+0000 7fefbccc3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:26:14.780 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.794+0000 7fefbccc3640 10 monclient(hunting): picked mon.noname-a con 0x7fefb81422b0 addr v1:172.21.15.86:6789/0 2024-03-29T11:26:14.780 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.794+0000 7fefbccc3640 10 monclient(hunting): picked mon.noname-b con 0x7fefb8142690 addr v1:172.21.15.86:6790/0 2024-03-29T11:26:14.780 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.794+0000 7fefbccc3640 10 monclient(hunting): picked mon.noname-c con 0x7fefb80b17d0 addr v1:172.21.15.196:6789/0 2024-03-29T11:26:14.780 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.794+0000 7fefbccc3640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb8142c80 con 0x7fefb81422b0 2024-03-29T11:26:14.780 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.794+0000 7fefbccc3640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb80aae40 con 0x7fefb80b17d0 2024-03-29T11:26:14.780 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.794+0000 7fefbccc3640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb813fbd0 con 0x7fefb8142690 2024-03-29T11:26:14.780 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.794+0000 7fefbccc3640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:14.780 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.794+0000 7fefbccc3640 10 monclient(hunting): authenticate will time out at 1199.112393s 2024-03-29T11:26:14.780 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.794+0000 7fefb5d9b640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7fefb8142690 0x7fefb80b2590 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:40496/0 (socket says 172.21.15.86:40496) 2024-03-29T11:26:14.780 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.794+0000 7fefb5d9b640 1 -- 172.21.15.86:0/4210491625 learned_addr learned my addr 172.21.15.86:0/4210491625 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:26:14.781 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.795+0000 7fefb559a640 1 -- 172.21.15.86:0/4210491625 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2547938080 0 0) 0x7fefb813fbd0 con 0x7fefb8142690 2024-03-29T11:26:14.781 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.795+0000 7fefb559a640 10 monclient(hunting): _negotiate 2024-03-29T11:26:14.781 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.795+0000 7fefb559a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:14.781 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.795+0000 7fefb559a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:14.781 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.795+0000 7fefb559a640 10 monclient(hunting): my global_id is 4460 2024-03-29T11:26:14.781 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.795+0000 7fefb559a640 1 -- 172.21.15.86:0/4210491625 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefa0003520 con 0x7fefb8142690 2024-03-29T11:26:14.782 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.795+0000 7fefb559a640 1 -- 172.21.15.86:0/4210491625 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1228272002 0 0) 0x7fefa0003520 con 0x7fefb8142690 2024-03-29T11:26:14.782 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.795+0000 7fefb559a640 1 -- 172.21.15.86:0/4210491625 >> v1:172.21.15.196:6789/0 conn(0x7fefb80b17d0 legacy=0x7fefb80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:14.782 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.795+0000 7fefb559a640 1 -- 172.21.15.86:0/4210491625 >> v1:172.21.15.86:6789/0 conn(0x7fefb81422b0 legacy=0x7fefb813f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:14.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.797+0000 7fefb559a640 10 monclient: _finish_hunting 0 2024-03-29T11:26:14.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.797+0000 7fefb559a640 1 monclient: found mon.noname-b 2024-03-29T11:26:14.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.797+0000 7fefb559a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:14.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.797+0000 7fefb559a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:26:14.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.797+0000 7fefb559a640 1 -- 172.21.15.86:0/4210491625 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb8151940 con 0x7fefb8142690 2024-03-29T11:26:14.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.797+0000 7fefb559a640 10 monclient: _finish_auth 0 2024-03-29T11:26:14.784 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.797+0000 7fefb559a640 10 monclient: _check_auth_tickets 2024-03-29T11:26:14.784 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.797+0000 7fefb559a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:14.784 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.797+0000 7fefb559a640 1 -- 172.21.15.86:0/4210491625 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fefac001690 con 0x7fefb8142690 2024-03-29T11:26:14.784 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.797+0000 7fefbccc3640 5 monclient: authenticate success, global_id 4460 2024-03-29T11:26:14.784 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.797+0000 7fefbccc3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:14.784 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.797+0000 7fefb559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:14.784 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.797+0000 7fefb559a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:14.784 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.797+0000 7fefb559a640 10 monclient: dump: 2024-03-29T11:26:14.784 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:14.784 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:14.784 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:14.784 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:14.784 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:14.784 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:14.784 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:14.785 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:14.785 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:14.785 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:14.785 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.797+0000 7fefbccc3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:14.785 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.797+0000 7fefb559a640 1 -- 172.21.15.86:0/4210491625 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fefac0017f0 con 0x7fefb8142690 2024-03-29T11:26:14.785 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.797+0000 7fefb559a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:14.785 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.797+0000 7fefb559a640 1 -- 172.21.15.86:0/4210491625 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fefac001c40 con 0x7fefb8142690 2024-03-29T11:26:14.785 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.797+0000 7fefb559a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:14.785 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.797+0000 7fefb559a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:14.785 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.797+0000 7fefb559a640 10 monclient: dump: 2024-03-29T11:26:14.785 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:14.785 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:14.785 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:14.785 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:14.785 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:14.785 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:14.786 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:14.786 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:14.786 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:14.786 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:14.786 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.797+0000 7fefbccc3640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:14.786 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.797+0000 7fefbccc3640 4 set_mon_vals no callback set 2024-03-29T11:26:14.789 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.798+0000 7fefbccc3640 10 monclient: shutdown 2024-03-29T11:26:14.789 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.798+0000 7fefbccc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:14.789 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.798+0000 7fefbccc3640 1 -- 172.21.15.86:0/4210491625 >> v1:172.21.15.86:6790/0 conn(0x7fefb8142690 legacy=0x7fefb80b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:14.789 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.798+0000 7fefbccc3640 1 -- 172.21.15.86:0/4210491625 shutdown_connections 2024-03-29T11:26:14.789 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.798+0000 7fefbccc3640 1 -- 172.21.15.86:0/4210491625 >> 172.21.15.86:0/4210491625 conn(0x7fefb8066a00 msgr2=0x7fefb8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:14.789 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.798+0000 7fefbccc3640 1 -- 172.21.15.86:0/4210491625 shutdown_connections 2024-03-29T11:26:14.789 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.798+0000 7fefbccc3640 1 -- 172.21.15.86:0/4210491625 wait complete. 2024-03-29T11:26:14.789 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.799+0000 7fefbccc3640 10 monclient: build_initial_monmap 2024-03-29T11:26:14.789 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.799+0000 7fefbccc3640 10 monclient: monmap: 2024-03-29T11:26:14.789 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:26:14.789 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:14.789 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:26:14.789 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:26:14.790 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:14.790 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:14.790 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:26:14.790 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:26:14.790 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:26:14.790 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:14.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.800+0000 7fefbccc3640 1 librados: starting msgr at 2024-03-29T11:26:14.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.800+0000 7fefbccc3640 1 librados: starting objecter 2024-03-29T11:26:14.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.800+0000 7fefbccc3640 1 Processor -- start 2024-03-29T11:26:14.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.800+0000 7fefbccc3640 1 -- start start 2024-03-29T11:26:14.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.800+0000 7fefbccc3640 1 librados: setting wanted keys 2024-03-29T11:26:14.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.800+0000 7fefbccc3640 1 librados: calling monclient init 2024-03-29T11:26:14.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.800+0000 7fefbccc3640 10 monclient: init 2024-03-29T11:26:14.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.801+0000 7fefbccc3640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:14.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.801+0000 7fefbccc3640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:26:14.791 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.801+0000 7fefbccc3640 10 monclient(hunting): picked mon.noname-a con 0x7fefb80b17d0 addr v1:172.21.15.86:6789/0 2024-03-29T11:26:14.791 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.801+0000 7fefbccc3640 10 monclient(hunting): picked mon.noname-c con 0x7fefb81422b0 addr v1:172.21.15.86:6790/0 2024-03-29T11:26:14.791 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.801+0000 7fefb5d9b640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7fefb81422b0 0x7fefb812ec40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:40832/0 (socket says 172.21.15.86:40832) 2024-03-29T11:26:14.791 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.802+0000 7fefb5d9b640 1 -- 172.21.15.86:0/4048420573 learned_addr learned my addr 172.21.15.86:0/4048420573 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:26:14.791 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.802+0000 7fefa6ffd640 10 client.?.objecter ms_handle_connect 0x7fefb81422b0 2024-03-29T11:26:14.791 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.802+0000 7fefa6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:14.791 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.802+0000 7fefbccc3640 10 monclient(hunting): picked mon.noname-b con 0x7fefb8142690 addr v1:172.21.15.196:6789/0 2024-03-29T11:26:14.791 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.802+0000 7fefbccc3640 1 -- 172.21.15.86:0/4048420573 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb80acdb0 con 0x7fefb80b17d0 2024-03-29T11:26:14.791 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.802+0000 7fefbccc3640 1 -- 172.21.15.86:0/4048420573 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb80ad090 con 0x7fefb8142690 2024-03-29T11:26:14.791 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.802+0000 7fefbccc3640 1 -- 172.21.15.86:0/4048420573 --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fefb80ab540 con 0x7fefb81422b0 2024-03-29T11:26:14.791 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.802+0000 7fefbccc3640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:14.791 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.802+0000 7fefbccc3640 10 monclient(hunting): authenticate will time out at 1199.120341s 2024-03-29T11:26:14.791 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.802+0000 7fefa6ffd640 1 -- 172.21.15.86:0/4048420573 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1153483434 0 0) 0x7fefb80ab540 con 0x7fefb81422b0 2024-03-29T11:26:14.791 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.802+0000 7fefa6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:14.791 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.802+0000 7fefa6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:14.792 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.802+0000 7fefa6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:14.792 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.802+0000 7fefa6ffd640 10 monclient(hunting): my global_id is 4463 2024-03-29T11:26:14.792 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.802+0000 7fefa6ffd640 1 -- 172.21.15.86:0/4048420573 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fef98004eb0 con 0x7fefb81422b0 2024-03-29T11:26:14.792 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.802+0000 7fefa6ffd640 1 -- 172.21.15.86:0/4048420573 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3648553971 0 0) 0x7fef98004eb0 con 0x7fefb81422b0 2024-03-29T11:26:14.792 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.802+0000 7fefa6ffd640 1 -- 172.21.15.86:0/4048420573 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fefb80ab540 con 0x7fefb81422b0 2024-03-29T11:26:14.792 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.802+0000 7fefa6ffd640 1 -- 172.21.15.86:0/4048420573 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fefac001c00 con 0x7fefb81422b0 2024-03-29T11:26:14.792 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.802+0000 7fefa6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:14.792 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.802+0000 7fefa6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:14.792 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.802+0000 7fefa6ffd640 10 monclient(hunting): dump: 2024-03-29T11:26:14.792 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:14.792 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:14.792 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:14.792 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:14.792 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:14.792 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:14.793 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:14.793 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:14.793 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:14.793 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:14.793 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.802+0000 7fefa6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:14.793 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.802+0000 7fefa6ffd640 1 -- 172.21.15.86:0/4048420573 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 903973360 0 0) 0x7fefb80ab540 con 0x7fefb81422b0 2024-03-29T11:26:14.793 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.802+0000 7fefa6ffd640 1 -- 172.21.15.86:0/4048420573 >> v1:172.21.15.196:6789/0 conn(0x7fefb8142690 legacy=0x7fefb81e2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:14.793 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.802+0000 7fefa6ffd640 1 -- 172.21.15.86:0/4048420573 >> v1:172.21.15.86:6789/0 conn(0x7fefb80b17d0 legacy=0x7fefb812fba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:26:14.795 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.804+0000 7fefa6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:14.795 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.804+0000 7fefa6ffd640 1 monclient: found mon.c 2024-03-29T11:26:14.795 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.804+0000 7fefa6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:14.795 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.804+0000 7fefa6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:26:14.795 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.804+0000 7fefa6ffd640 1 -- 172.21.15.86:0/4048420573 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb80ab820 con 0x7fefb81422b0 2024-03-29T11:26:14.795 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.804+0000 7fefa6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:26:14.795 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.804+0000 7fefa6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:14.795 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.804+0000 7fefa6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:14.795 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.804+0000 7fefbccc3640 5 monclient: authenticate success, global_id 4463 2024-03-29T11:26:14.795 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.804+0000 7fefbccc3640 10 monclient: _renew_subs 2024-03-29T11:26:14.795 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.804+0000 7fefbccc3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:26:14.795 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.804+0000 7fefbccc3640 1 -- 172.21.15.86:0/4048420573 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefb80abd40 con 0x7fefb81422b0 2024-03-29T11:26:14.795 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.804+0000 7fefbccc3640 10 client.4463.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:14.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.804+0000 7fefbccc3640 10 monclient: _renew_subs 2024-03-29T11:26:14.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.804+0000 7fefbccc3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:26:14.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.804+0000 7fefa6ffd640 1 -- 172.21.15.86:0/4048420573 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fefac001d60 con 0x7fefb81422b0 2024-03-29T11:26:14.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.804+0000 7fefbccc3640 1 -- 172.21.15.86:0/4048420573 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fefb80ac250 con 0x7fefb81422b0 2024-03-29T11:26:14.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.804+0000 7fefa6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:14.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.804+0000 7fefa6ffd640 1 -- 172.21.15.86:0/4048420573 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fefac004d30 con 0x7fefb81422b0 2024-03-29T11:26:14.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.804+0000 7fefa6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:14.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.804+0000 7fefa6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:14.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.804+0000 7fefb4d99640 4 set_mon_vals no callback set 2024-03-29T11:26:14.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.804+0000 7fefa6ffd640 10 monclient: dump: 2024-03-29T11:26:14.796 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:14.796 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:14.797 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:14.797 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:14.797 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:14.797 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:14.797 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:14.797 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:14.797 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:14.797 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:14.797 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.805+0000 7fefbccc3640 1 librados: init done 2024-03-29T11:26:14.797 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.806+0000 7fefa6ffd640 1 -- 172.21.15.86:0/4048420573 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7fefac004e90 con 0x7fefb81422b0 2024-03-29T11:26:14.797 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.806+0000 7fefa6ffd640 1 -- 172.21.15.86:0/4048420573 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7fefac084c60 con 0x7fefb81422b0 2024-03-29T11:26:14.797 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.806+0000 7fefa6ffd640 10 client.4463.objecter ms_dispatch 0x7fefb8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:14.797 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.806+0000 7fefa6ffd640 3 client.4463.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:14.797 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.806+0000 7fefa6ffd640 3 client.4463.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:14.798 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.806+0000 7fefb659c640 10 monclient: get_auth_request con 0x7fef9806dd10 auth_method 0 2024-03-29T11:26:14.798 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.806+0000 7fefa6ffd640 20 client.4463.objecter dump_active .. 0 homeless 2024-03-29T11:26:14.798 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.807+0000 7fefbccc3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:14.798 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.807+0000 7fefbccc3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:14.798 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.807+0000 7fefbccc3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:26:14.798 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.807+0000 7fefa6ffd640 10 client.4463.objecter ms_handle_connect 0x7fef9806dd10 2024-03-29T11:26:14.798 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.807+0000 7fefbccc3640 1 -- 172.21.15.86:0/4048420573 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefb80aba60 con 0x7fefb81422b0 2024-03-29T11:26:14.798 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.812+0000 7fefa6ffd640 1 -- 172.21.15.86:0/4048420573 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fefb80aba60 con 0x7fefb81422b0 2024-03-29T11:26:14.798 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.812+0000 7fefa6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:14.799 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:14.812+0000 7fefa6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:15.080 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:15.094+0000 7fefbccc3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:15.080 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:15.094+0000 7fefbccc3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:15.080 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:15.094+0000 7fefbccc3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:26:15.081 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:15.094+0000 7fefbccc3640 1 -- 172.21.15.86:0/4048420573 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fefb80aba60 con 0x7fefb81422b0 2024-03-29T11:26:15.081 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:15.095+0000 7fefa6ffd640 1 -- 172.21.15.86:0/4048420573 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7fefb80aba60 con 0x7fefb81422b0 2024-03-29T11:26:15.081 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:15.095+0000 7fefa6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:15.081 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:15.095+0000 7fefa6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:15.081 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:26:15.081 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:15.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:15.098+0000 7fefbccc3640 10 librados: watch_flush enter 2024-03-29T11:26:15.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:15.098+0000 7fefbccc3640 10 librados: watch_flush exit 2024-03-29T11:26:15.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:15.098+0000 7fefbccc3640 20 client.4463.objecter shutdown clearing up homeless session... 2024-03-29T11:26:15.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:15.098+0000 7fefbccc3640 10 client.4463.objecter successfully canceled tick 2024-03-29T11:26:15.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:15.098+0000 7fefbccc3640 1 -- 172.21.15.86:0/4048420573 >> v1:172.21.15.196:6816/588110318 conn(0x7fef9806dd10 legacy=0x7fef980701d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:15.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:15.098+0000 7fefbccc3640 10 monclient: shutdown 2024-03-29T11:26:15.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:15.098+0000 7fefbccc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:15.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:15.098+0000 7fefbccc3640 1 -- 172.21.15.86:0/4048420573 >> v1:172.21.15.86:6790/0 conn(0x7fefb81422b0 legacy=0x7fefb812ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:15.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:15.098+0000 7fefbccc3640 1 -- 172.21.15.86:0/4048420573 shutdown_connections 2024-03-29T11:26:15.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:15.098+0000 7fefbccc3640 1 -- 172.21.15.86:0/4048420573 >> 172.21.15.86:0/4048420573 conn(0x7fefb8066a00 msgr2=0x7fefb80b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:15.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:15.098+0000 7fefbccc3640 1 -- 172.21.15.86:0/4048420573 shutdown_connections 2024-03-29T11:26:15.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:15.098+0000 7fefbccc3640 1 -- 172.21.15.86:0/4048420573 wait complete. 2024-03-29T11:26:15.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:15.098+0000 7fefbccc3640 1 librados: shutdown 2024-03-29T11:26:15.099 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:21.907 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:22.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.060+0000 7f1ecb51c640 10 monclient: get_monmap_and_config 2024-03-29T11:26:22.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.060+0000 7f1ecb51c640 10 monclient: build_initial_monmap 2024-03-29T11:26:22.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.060+0000 7f1ecb51c640 10 monclient: monmap: 2024-03-29T11:26:22.046 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:26:22.046 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:22.047 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:26:22.061014+0000 2024-03-29T11:26:22.047 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:26:22.061014+0000 2024-03-29T11:26:22.047 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:22.047 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:22.047 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:26:22.047 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:26:22.047 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:26:22.047 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:22.047 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.060+0000 7f1ecb51c640 1 Processor -- start 2024-03-29T11:26:22.049 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.061+0000 7f1ecb51c640 1 -- start start 2024-03-29T11:26:22.049 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.061+0000 7f1ecb51c640 10 monclient: init 2024-03-29T11:26:22.049 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.061+0000 7f1ecb51c640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:22.049 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.061+0000 7f1ecb51c640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:26:22.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.061+0000 7f1ecb51c640 10 monclient(hunting): picked mon.noname-a con 0x7f1ec40a9750 addr v1:172.21.15.86:6789/0 2024-03-29T11:26:22.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.061+0000 7f1ecb51c640 10 monclient(hunting): picked mon.noname-b con 0x7f1ec413e1e0 addr v1:172.21.15.86:6790/0 2024-03-29T11:26:22.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.062+0000 7f1ecb51c640 10 monclient(hunting): picked mon.noname-c con 0x7f1ec4071c30 addr v1:172.21.15.196:6789/0 2024-03-29T11:26:22.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.062+0000 7f1ecb51c640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ec40ab300 con 0x7f1ec40a9750 2024-03-29T11:26:22.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.062+0000 7f1ecb51c640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ec40ac2f0 con 0x7f1ec4071c30 2024-03-29T11:26:22.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.062+0000 7f1ecb51c640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ec40ad8d0 con 0x7f1ec413e1e0 2024-03-29T11:26:22.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.062+0000 7f1ec9ab9640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f1ec4071c30 0x7f1ec40ace20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:49938/0 (socket says 172.21.15.86:49938) 2024-03-29T11:26:22.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.062+0000 7f1ec9ab9640 1 -- 172.21.15.86:0/1436580511 learned_addr learned my addr 172.21.15.86:0/1436580511 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:26:22.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.062+0000 7f1ecb51c640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:22.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.062+0000 7f1ecb51c640 10 monclient(hunting): authenticate will time out at 1206.380395s 2024-03-29T11:26:22.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.062+0000 7f1ebbfff640 1 -- 172.21.15.86:0/1436580511 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2708963201 0 0) 0x7f1ec40ac2f0 con 0x7f1ec4071c30 2024-03-29T11:26:22.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.062+0000 7f1ebbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:22.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.062+0000 7f1ebbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:22.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.062+0000 7f1ebbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:22.051 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.062+0000 7f1ebbfff640 10 monclient(hunting): my global_id is 4405 2024-03-29T11:26:22.051 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.062+0000 7f1ebbfff640 1 -- 172.21.15.86:0/1436580511 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1eb0003440 con 0x7f1ec4071c30 2024-03-29T11:26:22.051 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.063+0000 7f1ebbfff640 1 -- 172.21.15.86:0/1436580511 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2211410007 0 0) 0x7f1eb0003440 con 0x7f1ec4071c30 2024-03-29T11:26:22.051 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.063+0000 7f1ebbfff640 1 -- 172.21.15.86:0/1436580511 >> v1:172.21.15.86:6790/0 conn(0x7f1ec413e1e0 legacy=0x7f1ec40ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:22.051 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.063+0000 7f1ebbfff640 1 -- 172.21.15.86:0/1436580511 >> v1:172.21.15.86:6789/0 conn(0x7f1ec40a9750 legacy=0x7f1ec413dad0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:22.051 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.065+0000 7f1ebbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:22.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.065+0000 7f1ebbfff640 1 monclient: found mon.noname-c 2024-03-29T11:26:22.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.065+0000 7f1ebbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:22.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.065+0000 7f1ebbfff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T11:26:22.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.065+0000 7f1ebbfff640 1 -- 172.21.15.86:0/1436580511 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec40ada60 con 0x7f1ec4071c30 2024-03-29T11:26:22.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.065+0000 7f1ebbfff640 10 monclient: _finish_auth 0 2024-03-29T11:26:22.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.065+0000 7f1ebbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:22.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.065+0000 7f1ebbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:22.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.065+0000 7f1ecb51c640 5 monclient: authenticate success, global_id 4405 2024-03-29T11:26:22.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.065+0000 7f1ebbfff640 1 -- 172.21.15.86:0/1436580511 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f1ebc001a10 con 0x7f1ec4071c30 2024-03-29T11:26:22.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.065+0000 7f1ebbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:22.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.065+0000 7f1ebbfff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:22.053 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.065+0000 7f1ebbfff640 10 monclient: dump: 2024-03-29T11:26:22.053 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:22.053 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:22.053 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:22.053 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:22.053 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:22.053 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:22.053 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:22.053 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:22.053 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:22.054 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:22.054 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.065+0000 7f1ecb51c640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:22.054 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.065+0000 7f1ebbfff640 1 -- 172.21.15.86:0/1436580511 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ebc001be0 con 0x7f1ec4071c30 2024-03-29T11:26:22.054 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.065+0000 7f1ebbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:22.054 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.065+0000 7f1ebbfff640 1 -- 172.21.15.86:0/1436580511 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f1ebc002030 con 0x7f1ec4071c30 2024-03-29T11:26:22.054 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.065+0000 7f1ebbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:22.054 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.065+0000 7f1ebbfff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:22.054 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.065+0000 7f1ebbfff640 10 monclient: dump: 2024-03-29T11:26:22.054 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:22.054 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:22.054 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:22.054 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:22.054 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:22.054 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:22.054 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:22.055 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:22.055 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:22.055 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:22.055 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.065+0000 7f1ecb51c640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:22.055 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.065+0000 7f1ecb51c640 4 set_mon_vals no callback set 2024-03-29T11:26:22.056 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.066+0000 7f1ecb51c640 10 monclient: shutdown 2024-03-29T11:26:22.056 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.066+0000 7f1ecb51c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:22.057 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.066+0000 7f1ecb51c640 1 -- 172.21.15.86:0/1436580511 >> v1:172.21.15.196:6789/0 conn(0x7f1ec4071c30 legacy=0x7f1ec40ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:22.057 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.066+0000 7f1ecb51c640 1 -- 172.21.15.86:0/1436580511 shutdown_connections 2024-03-29T11:26:22.057 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.066+0000 7f1ecb51c640 1 -- 172.21.15.86:0/1436580511 >> 172.21.15.86:0/1436580511 conn(0x7f1ec406b660 msgr2=0x7f1ec406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:22.057 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.067+0000 7f1ecb51c640 1 -- 172.21.15.86:0/1436580511 shutdown_connections 2024-03-29T11:26:22.057 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.067+0000 7f1ecb51c640 1 -- 172.21.15.86:0/1436580511 wait complete. 2024-03-29T11:26:22.057 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.068+0000 7f1ecb51c640 10 monclient: build_initial_monmap 2024-03-29T11:26:22.057 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.068+0000 7f1ecb51c640 10 monclient: monmap: 2024-03-29T11:26:22.057 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:26:22.057 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:22.057 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:26:22.057 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:26:22.057 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:22.057 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:22.057 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:26:22.058 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:26:22.058 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:26:22.058 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:22.058 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.068+0000 7f1ecb51c640 1 librados: starting msgr at 2024-03-29T11:26:22.058 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.068+0000 7f1ecb51c640 1 librados: starting objecter 2024-03-29T11:26:22.058 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.069+0000 7f1ecb51c640 1 Processor -- start 2024-03-29T11:26:22.058 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.069+0000 7f1ecb51c640 1 -- start start 2024-03-29T11:26:22.058 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.069+0000 7f1ecb51c640 1 librados: setting wanted keys 2024-03-29T11:26:22.058 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.069+0000 7f1ecb51c640 1 librados: calling monclient init 2024-03-29T11:26:22.058 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.069+0000 7f1ecb51c640 10 monclient: init 2024-03-29T11:26:22.058 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.070+0000 7f1ecb51c640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:22.058 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.070+0000 7f1ecb51c640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:22.058 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.070+0000 7f1ecb51c640 10 monclient(hunting): picked mon.noname-c con 0x7f1ec4071c30 addr v1:172.21.15.86:6790/0 2024-03-29T11:26:22.064 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.070+0000 7f1ec92b8640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f1ec4071c30 0x7f1ec4133850 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:39648/0 (socket says 172.21.15.86:39648) 2024-03-29T11:26:22.064 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.070+0000 7f1ecb51c640 10 monclient(hunting): picked mon.noname-a con 0x7f1ec40a9750 addr v1:172.21.15.86:6789/0 2024-03-29T11:26:22.064 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.070+0000 7f1ec92b8640 1 -- 172.21.15.86:0/3353514613 learned_addr learned my addr 172.21.15.86:0/3353514613 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:26:22.064 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.071+0000 7f1ecb51c640 10 monclient(hunting): picked mon.noname-b con 0x7f1ec413e1e0 addr v1:172.21.15.196:6789/0 2024-03-29T11:26:22.065 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.071+0000 7f1ecb51c640 1 -- 172.21.15.86:0/3353514613 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ec4130400 con 0x7f1ec40a9750 2024-03-29T11:26:22.065 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.071+0000 7f1ecb51c640 1 -- 172.21.15.86:0/3353514613 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ec41320f0 con 0x7f1ec413e1e0 2024-03-29T11:26:22.065 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.071+0000 7f1eb9ffb640 10 client.?.objecter ms_handle_connect 0x7f1ec4071c30 2024-03-29T11:26:22.065 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.071+0000 7f1eb9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:22.065 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.071+0000 7f1ecb51c640 1 -- 172.21.15.86:0/3353514613 --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ec4132280 con 0x7f1ec4071c30 2024-03-29T11:26:22.065 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.071+0000 7f1ecb51c640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:22.065 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.071+0000 7f1ecb51c640 10 monclient(hunting): authenticate will time out at 1206.389324s 2024-03-29T11:26:22.065 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.071+0000 7f1eb9ffb640 10 client.?.objecter ms_handle_connect 0x7f1ec413e1e0 2024-03-29T11:26:22.065 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.071+0000 7f1eb9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:22.065 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.072+0000 7f1eb9ffb640 1 -- 172.21.15.86:0/3353514613 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3670822452 0 0) 0x7f1ec41320f0 con 0x7f1ec413e1e0 2024-03-29T11:26:22.065 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.072+0000 7f1eb9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:22.065 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.072+0000 7f1eb9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:22.065 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.072+0000 7f1eb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:22.065 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.072+0000 7f1eb9ffb640 10 monclient(hunting): my global_id is 4408 2024-03-29T11:26:22.066 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.072+0000 7f1eb9ffb640 1 -- 172.21.15.86:0/3353514613 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ea8004eb0 con 0x7f1ec413e1e0 2024-03-29T11:26:22.066 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.072+0000 7f1eb9ffb640 1 -- 172.21.15.86:0/3353514613 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2843395375 0 0) 0x7f1ea8004eb0 con 0x7f1ec413e1e0 2024-03-29T11:26:22.066 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.072+0000 7f1eb9ffb640 1 -- 172.21.15.86:0/3353514613 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ec41320f0 con 0x7f1ec413e1e0 2024-03-29T11:26:22.066 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.072+0000 7f1eb9ffb640 1 -- 172.21.15.86:0/3353514613 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f1ebc001ef0 con 0x7f1ec413e1e0 2024-03-29T11:26:22.066 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.072+0000 7f1eb9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:22.066 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.072+0000 7f1eb9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:22.066 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.072+0000 7f1eb9ffb640 10 monclient(hunting): dump: 2024-03-29T11:26:22.066 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:22.066 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:22.066 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:22.066 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:22.066 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:22.066 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:22.066 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:22.067 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:22.067 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:22.067 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:22.067 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.072+0000 7f1eb9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:22.067 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.073+0000 7f1eb9ffb640 1 -- 172.21.15.86:0/3353514613 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1697337067 0 0) 0x7f1ec4132280 con 0x7f1ec4071c30 2024-03-29T11:26:22.067 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.073+0000 7f1eb9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:22.067 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.073+0000 7f1eb9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:22.067 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.073+0000 7f1eb9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:22.067 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.073+0000 7f1eb9ffb640 10 monclient(hunting): my global_id is 4466 2024-03-29T11:26:22.067 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.073+0000 7f1eb9ffb640 1 -- 172.21.15.86:0/3353514613 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ea8004eb0 con 0x7f1ec4071c30 2024-03-29T11:26:22.067 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.073+0000 7f1eb9ffb640 1 -- 172.21.15.86:0/3353514613 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 327845679 0 0) 0x7f1ea8004eb0 con 0x7f1ec4071c30 2024-03-29T11:26:22.067 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.073+0000 7f1eb9ffb640 1 -- 172.21.15.86:0/3353514613 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f1ec4132280 con 0x7f1ec4071c30 2024-03-29T11:26:22.067 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.073+0000 7f1eb9ffb640 1 -- 172.21.15.86:0/3353514613 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f1ec0003fc0 con 0x7f1ec4071c30 2024-03-29T11:26:22.067 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.073+0000 7f1eb9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:22.067 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.073+0000 7f1eb9ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:22.068 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.073+0000 7f1eb9ffb640 10 monclient(hunting): dump: 2024-03-29T11:26:22.068 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:22.068 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:22.068 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:22.068 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:22.068 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:22.068 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:22.068 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:22.068 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:22.068 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:22.068 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:22.068 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.073+0000 7f1eb9ffb640 1 -- 172.21.15.86:0/3353514613 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2243463530 0 0) 0x7f1ec41320f0 con 0x7f1ec413e1e0 2024-03-29T11:26:22.068 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.073+0000 7f1eb9ffb640 1 -- 172.21.15.86:0/3353514613 >> v1:172.21.15.86:6790/0 conn(0x7f1ec4071c30 legacy=0x7f1ec4133850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:22.069 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.073+0000 7f1eb9ffb640 1 -- 172.21.15.86:0/3353514613 >> v1:172.21.15.86:6789/0 conn(0x7f1ec40a9750 legacy=0x7f1ec4133f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:22.069 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.073+0000 7f1eb9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:26:22.069 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.073+0000 7f1eb9ffb640 1 monclient: found mon.b 2024-03-29T11:26:22.069 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.073+0000 7f1eb9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:22.069 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.073+0000 7f1eb9ffb640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:26:22.069 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.073+0000 7f1eb9ffb640 1 -- 172.21.15.86:0/3353514613 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec412fe90 con 0x7f1ec413e1e0 2024-03-29T11:26:22.069 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.073+0000 7f1eb9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:26:22.069 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.073+0000 7f1eb9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:26:22.069 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.073+0000 7f1eb9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:22.069 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.073+0000 7f1ecb51c640 5 monclient: authenticate success, global_id 4408 2024-03-29T11:26:22.069 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.073+0000 7f1ecb51c640 10 monclient: _renew_subs 2024-03-29T11:26:22.069 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.073+0000 7f1ecb51c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:26:22.069 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.073+0000 7f1ecb51c640 1 -- 172.21.15.86:0/3353514613 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ec41300e0 con 0x7f1ec413e1e0 2024-03-29T11:26:22.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.074+0000 7f1ecb51c640 10 client.4408.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:22.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.074+0000 7f1ecb51c640 10 monclient: _renew_subs 2024-03-29T11:26:22.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.074+0000 7f1ecb51c640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:26:22.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.074+0000 7f1ecb51c640 1 -- 172.21.15.86:0/3353514613 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ec412e300 con 0x7f1ec413e1e0 2024-03-29T11:26:22.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.074+0000 7f1ecb51c640 1 librados: init done 2024-03-29T11:26:22.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.075+0000 7f1eaf7fe640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:22.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.075+0000 7f1eaf7fe640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:22.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.075+0000 7f1eaf7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:26:22.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.075+0000 7f1eaf7fe640 1 -- 172.21.15.86:0/3353514613 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ec4075f10 con 0x7f1ec413e1e0 2024-03-29T11:26:22.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.076+0000 7f1eb9ffb640 1 -- 172.21.15.86:0/3353514613 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1ebc0020f0 con 0x7f1ec413e1e0 2024-03-29T11:26:22.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.077+0000 7f1eb9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:22.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.077+0000 7f1eb9ffb640 1 -- 172.21.15.86:0/3353514613 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f1ebc001390 con 0x7f1ec413e1e0 2024-03-29T11:26:22.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.077+0000 7f1eb9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:22.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.077+0000 7f1eb9ffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:22.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.077+0000 7f1eb9ffb640 10 monclient: dump: 2024-03-29T11:26:22.071 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:22.071 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:22.071 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:22.071 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:22.071 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:22.071 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:22.071 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:22.071 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:22.071 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:22.071 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:22.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.077+0000 7f1ebb7fe640 4 set_mon_vals no callback set 2024-03-29T11:26:22.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.079+0000 7f1eb9ffb640 1 -- 172.21.15.86:0/3353514613 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f1ebc01af80 con 0x7f1ec413e1e0 2024-03-29T11:26:22.073 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.079+0000 7f1eb9ffb640 1 -- 172.21.15.86:0/3353514613 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f1ebc084e60 con 0x7f1ec413e1e0 2024-03-29T11:26:22.073 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.079+0000 7f1eb9ffb640 10 client.4408.objecter ms_dispatch 0x7f1ec4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:22.073 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.079+0000 7f1eb9ffb640 3 client.4408.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:22.073 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.079+0000 7f1eb9ffb640 3 client.4408.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:22.074 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.079+0000 7f1eb9ffb640 20 client.4408.objecter dump_active .. 0 homeless 2024-03-29T11:26:22.074 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.080+0000 7f1ec92b8640 10 monclient: get_auth_request con 0x7f1ea80765a0 auth_method 0 2024-03-29T11:26:22.074 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.081+0000 7f1eb9ffb640 10 client.4408.objecter ms_handle_connect 0x7f1ea80765a0 2024-03-29T11:26:22.074 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.086+0000 7f1eb9ffb640 1 -- 172.21.15.86:0/3353514613 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1ebc050740 con 0x7f1ec413e1e0 2024-03-29T11:26:22.074 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.086+0000 7f1eb9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:22.074 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.086+0000 7f1eb9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:22.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.393+0000 7f1eaf7fe640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:22.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.393+0000 7f1eaf7fe640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:22.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.393+0000 7f1eaf7fe640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:26:22.380 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:26:22.380 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:22.381 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.393+0000 7f1eaf7fe640 1 -- 172.21.15.86:0/3353514613 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ec4071a10 con 0x7f1ec413e1e0 2024-03-29T11:26:22.381 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.394+0000 7f1eb9ffb640 1 -- 172.21.15.86:0/3353514613 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f1ebc052670 con 0x7f1ec413e1e0 2024-03-29T11:26:22.381 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.394+0000 7f1eb9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:22.381 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.394+0000 7f1eb9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:22.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.398+0000 7f1eaf7fe640 10 librados: watch_flush enter 2024-03-29T11:26:22.384 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.398+0000 7f1eaf7fe640 10 librados: watch_flush exit 2024-03-29T11:26:22.384 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.398+0000 7f1eaf7fe640 20 client.4408.objecter shutdown clearing up homeless session... 2024-03-29T11:26:22.384 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.398+0000 7f1eaf7fe640 10 client.4408.objecter successfully canceled tick 2024-03-29T11:26:22.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.399+0000 7f1eaf7fe640 1 -- 172.21.15.86:0/3353514613 >> v1:172.21.15.196:6816/588110318 conn(0x7f1ea80765a0 legacy=0x7f1ea8078a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:22.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.399+0000 7f1eaf7fe640 10 monclient: shutdown 2024-03-29T11:26:22.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.399+0000 7f1eaf7fe640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:22.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.399+0000 7f1eaf7fe640 1 -- 172.21.15.86:0/3353514613 >> v1:172.21.15.196:6789/0 conn(0x7f1ec413e1e0 legacy=0x7f1ec41e1a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:22.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.399+0000 7f1eaf7fe640 1 -- 172.21.15.86:0/3353514613 shutdown_connections 2024-03-29T11:26:22.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.399+0000 7f1eaf7fe640 1 -- 172.21.15.86:0/3353514613 >> 172.21.15.86:0/3353514613 conn(0x7f1ec406b660 msgr2=0x7f1ec40ae120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:22.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.400+0000 7f1eaf7fe640 1 -- 172.21.15.86:0/3353514613 shutdown_connections 2024-03-29T11:26:22.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.400+0000 7f1eaf7fe640 1 -- 172.21.15.86:0/3353514613 wait complete. 2024-03-29T11:26:22.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:22.400+0000 7f1eaf7fe640 1 librados: shutdown 2024-03-29T11:26:22.399 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:29.206 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:29.348 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.360+0000 7f817ba01640 10 monclient: get_monmap_and_config 2024-03-29T11:26:29.348 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.360+0000 7f817ba01640 10 monclient: build_initial_monmap 2024-03-29T11:26:29.348 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.360+0000 7f817ba01640 10 monclient: monmap: 2024-03-29T11:26:29.348 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:26:29.349 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:29.349 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:26:29.361769+0000 2024-03-29T11:26:29.349 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:26:29.361769+0000 2024-03-29T11:26:29.349 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:29.349 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:29.349 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:26:29.349 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:26:29.349 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:26:29.349 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:29.349 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.361+0000 7f817ba01640 1 Processor -- start 2024-03-29T11:26:29.349 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.362+0000 7f817ba01640 1 -- start start 2024-03-29T11:26:29.349 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.362+0000 7f817ba01640 10 monclient: init 2024-03-29T11:26:29.351 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.363+0000 7f817ba01640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:29.351 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.363+0000 7f817ba01640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:29.351 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.363+0000 7f817ba01640 10 monclient(hunting): picked mon.noname-b con 0x7f81740a9ac0 addr v1:172.21.15.86:6790/0 2024-03-29T11:26:29.351 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.363+0000 7f817ba01640 10 monclient(hunting): picked mon.noname-a con 0x7f817413ee90 addr v1:172.21.15.86:6789/0 2024-03-29T11:26:29.351 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.363+0000 7f817ba01640 10 monclient(hunting): picked mon.noname-c con 0x7f81740b4100 addr v1:172.21.15.196:6789/0 2024-03-29T11:26:29.351 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.363+0000 7f817ba01640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81740af6f0 con 0x7f817413ee90 2024-03-29T11:26:29.351 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.363+0000 7f817ba01640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81740aac80 con 0x7f81740b4100 2024-03-29T11:26:29.351 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.363+0000 7f817ba01640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81740aaf60 con 0x7f81740a9ac0 2024-03-29T11:26:29.351 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.363+0000 7f817ba01640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:29.352 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.363+0000 7f817ba01640 10 monclient(hunting): authenticate will time out at 1213.681145s 2024-03-29T11:26:29.355 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.366+0000 7f817979d640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f81740a9ac0 0x7f81740adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:41666/0 (socket says 172.21.15.86:41666) 2024-03-29T11:26:29.356 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.366+0000 7f817979d640 1 -- 172.21.15.86:0/1445544675 learned_addr learned my addr 172.21.15.86:0/1445544675 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:26:29.356 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.366+0000 7f816bfff640 1 -- 172.21.15.86:0/1445544675 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3465316465 0 0) 0x7f81740af6f0 con 0x7f817413ee90 2024-03-29T11:26:29.356 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.366+0000 7f816bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:29.356 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.366+0000 7f816bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:29.356 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.366+0000 7f816bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:29.356 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.366+0000 7f816bfff640 10 monclient(hunting): my global_id is 4419 2024-03-29T11:26:29.356 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.366+0000 7f816bfff640 1 -- 172.21.15.86:0/1445544675 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8160003520 con 0x7f817413ee90 2024-03-29T11:26:29.356 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.366+0000 7f816bfff640 1 -- 172.21.15.86:0/1445544675 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3700276687 0 0) 0x7f81740aaf60 con 0x7f81740a9ac0 2024-03-29T11:26:29.356 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.366+0000 7f816bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:29.356 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.366+0000 7f816bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:29.356 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.366+0000 7f816bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:29.356 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.366+0000 7f816bfff640 10 monclient(hunting): my global_id is 4469 2024-03-29T11:26:29.356 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.366+0000 7f816bfff640 1 -- 172.21.15.86:0/1445544675 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81740af6f0 con 0x7f81740a9ac0 2024-03-29T11:26:29.357 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.367+0000 7f816bfff640 1 -- 172.21.15.86:0/1445544675 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1205777331 0 0) 0x7f81740aac80 con 0x7f81740b4100 2024-03-29T11:26:29.357 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.367+0000 7f816bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:29.357 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.367+0000 7f816bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:29.357 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.367+0000 7f816bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:29.357 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.367+0000 7f816bfff640 10 monclient(hunting): my global_id is 4411 2024-03-29T11:26:29.357 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.367+0000 7f816bfff640 1 -- 172.21.15.86:0/1445544675 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81740aaf60 con 0x7f81740b4100 2024-03-29T11:26:29.357 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.367+0000 7f816bfff640 1 -- 172.21.15.86:0/1445544675 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1164720285 0 0) 0x7f8160003520 con 0x7f817413ee90 2024-03-29T11:26:29.357 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.367+0000 7f816bfff640 1 -- 172.21.15.86:0/1445544675 >> v1:172.21.15.86:6790/0 conn(0x7f81740a9ac0 legacy=0x7f81740adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:29.357 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.367+0000 7f816bfff640 1 -- 172.21.15.86:0/1445544675 >> v1:172.21.15.196:6789/0 conn(0x7f81740b4100 legacy=0x7f817414a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:29.357 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.367+0000 7f816bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:29.357 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.367+0000 7f816bfff640 1 monclient: found mon.noname-a 2024-03-29T11:26:29.357 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.367+0000 7f816bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:29.357 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.367+0000 7f816bfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.86:6789/0 2024-03-29T11:26:29.358 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.367+0000 7f816bfff640 1 -- 172.21.15.86:0/1445544675 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f817413e970 con 0x7f817413ee90 2024-03-29T11:26:29.358 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.367+0000 7f816bfff640 10 monclient: _finish_auth 0 2024-03-29T11:26:29.358 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.367+0000 7f816bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:29.358 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.367+0000 7f816bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:29.358 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.367+0000 7f816bfff640 1 -- 172.21.15.86:0/1445544675 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f8164001690 con 0x7f817413ee90 2024-03-29T11:26:29.358 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.367+0000 7f816bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:29.358 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.367+0000 7f816bfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:29.358 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.367+0000 7f816bfff640 10 monclient: dump: 2024-03-29T11:26:29.358 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:29.358 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:29.358 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:29.358 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:29.358 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:29.359 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:29.359 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:29.359 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:29.359 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:29.359 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:29.359 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.367+0000 7f817ba01640 5 monclient: authenticate success, global_id 4419 2024-03-29T11:26:29.359 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.367+0000 7f817ba01640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:29.359 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.367+0000 7f816bfff640 1 -- 172.21.15.86:0/1445544675 <== mon.0 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81640017f0 con 0x7f817413ee90 2024-03-29T11:26:29.359 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.367+0000 7f816bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:29.359 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.367+0000 7f817ba01640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:29.359 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.367+0000 7f817ba01640 4 set_mon_vals no callback set 2024-03-29T11:26:29.359 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.367+0000 7f816bfff640 1 -- 172.21.15.86:0/1445544675 <== mon.0 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f8164001c00 con 0x7f817413ee90 2024-03-29T11:26:29.359 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.367+0000 7f816bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:29.359 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.367+0000 7f816bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:29.360 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.367+0000 7f816bfff640 10 monclient: dump: 2024-03-29T11:26:29.360 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:29.360 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:29.360 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:29.360 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:29.360 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:29.360 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:29.360 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:29.360 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:29.360 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:29.360 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:29.360 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.368+0000 7f817ba01640 10 monclient: shutdown 2024-03-29T11:26:29.360 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.368+0000 7f817ba01640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:29.360 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.368+0000 7f817ba01640 1 -- 172.21.15.86:0/1445544675 >> v1:172.21.15.86:6789/0 conn(0x7f817413ee90 legacy=0x7f817413b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:29.361 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.368+0000 7f817ba01640 1 -- 172.21.15.86:0/1445544675 shutdown_connections 2024-03-29T11:26:29.361 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.368+0000 7f817ba01640 1 -- 172.21.15.86:0/1445544675 >> 172.21.15.86:0/1445544675 conn(0x7f81740b35e0 msgr2=0x7f81740b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:29.361 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.369+0000 7f817ba01640 1 -- 172.21.15.86:0/1445544675 shutdown_connections 2024-03-29T11:26:29.361 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.370+0000 7f817ba01640 1 -- 172.21.15.86:0/1445544675 wait complete. 2024-03-29T11:26:29.362 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.371+0000 7f817ba01640 10 monclient: build_initial_monmap 2024-03-29T11:26:29.362 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.371+0000 7f817ba01640 10 monclient: monmap: 2024-03-29T11:26:29.362 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:26:29.362 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:29.362 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:26:29.362 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:26:29.362 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:29.362 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:29.362 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:26:29.362 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:26:29.363 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:26:29.363 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:29.363 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.372+0000 7f817ba01640 1 librados: starting msgr at 2024-03-29T11:26:29.363 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.372+0000 7f817ba01640 1 librados: starting objecter 2024-03-29T11:26:29.363 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.372+0000 7f817ba01640 1 Processor -- start 2024-03-29T11:26:29.363 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.373+0000 7f817ba01640 1 -- start start 2024-03-29T11:26:29.363 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.373+0000 7f817ba01640 1 librados: setting wanted keys 2024-03-29T11:26:29.363 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.373+0000 7f817ba01640 1 librados: calling monclient init 2024-03-29T11:26:29.363 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.373+0000 7f817ba01640 10 monclient: init 2024-03-29T11:26:29.363 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.374+0000 7f817ba01640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:29.363 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.374+0000 7f817ba01640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:29.363 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.374+0000 7f817ba01640 10 monclient(hunting): picked mon.noname-c con 0x7f81740a9ac0 addr v1:172.21.15.86:6790/0 2024-03-29T11:26:29.363 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.374+0000 7f817ba01640 10 monclient(hunting): picked mon.noname-b con 0x7f81740b4100 addr v1:172.21.15.196:6789/0 2024-03-29T11:26:29.363 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.374+0000 7f817ba01640 10 monclient(hunting): picked mon.noname-a con 0x7f817413ee90 addr v1:172.21.15.86:6789/0 2024-03-29T11:26:29.364 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.374+0000 7f817ba01640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8174071d00 con 0x7f817413ee90 2024-03-29T11:26:29.364 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.374+0000 7f817ba01640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8174071f30 con 0x7f81740b4100 2024-03-29T11:26:29.364 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.374+0000 7f817ba01640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81740702c0 con 0x7f81740a9ac0 2024-03-29T11:26:29.364 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.374+0000 7f817ba01640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:29.364 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.374+0000 7f817ba01640 10 monclient(hunting): authenticate will time out at 1213.692543s 2024-03-29T11:26:29.377 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.377+0000 7f8178f9c640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f81740b4100 0x7f8174134730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:60260/0 (socket says 172.21.15.86:60260) 2024-03-29T11:26:29.377 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.377+0000 7f8178f9c640 1 -- 172.21.15.86:0/3508716901 learned_addr learned my addr 172.21.15.86:0/3508716901 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:26:29.378 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.377+0000 7f8169ffb640 10 client.?.objecter ms_handle_connect 0x7f81740b4100 2024-03-29T11:26:29.378 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.377+0000 7f8169ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:29.378 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.377+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2409796799 0 0) 0x7f8174071f30 con 0x7f81740b4100 2024-03-29T11:26:29.378 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.377+0000 7f8169ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:29.378 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.377+0000 7f8169ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:29.378 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.377+0000 7f8169ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:29.378 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.377+0000 7f8169ffb640 10 monclient(hunting): my global_id is 4414 2024-03-29T11:26:29.378 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.377+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8158004eb0 con 0x7f81740b4100 2024-03-29T11:26:29.378 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.378+0000 7f8169ffb640 10 client.?.objecter ms_handle_connect 0x7f81740a9ac0 2024-03-29T11:26:29.378 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.378+0000 7f8169ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:29.378 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.378+0000 7f8169ffb640 10 client.?.objecter ms_handle_connect 0x7f817413ee90 2024-03-29T11:26:29.378 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.378+0000 7f8169ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:29.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.378+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3177624450 0 0) 0x7f8158004eb0 con 0x7f81740b4100 2024-03-29T11:26:29.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.378+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8174071f30 con 0x7f81740b4100 2024-03-29T11:26:29.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.378+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2788568380 0 0) 0x7f8174071d00 con 0x7f817413ee90 2024-03-29T11:26:29.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.378+0000 7f8169ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:29.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.378+0000 7f8169ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:29.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.378+0000 7f8169ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:29.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.378+0000 7f8169ffb640 10 monclient(hunting): my global_id is 4422 2024-03-29T11:26:29.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.378+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8158004eb0 con 0x7f817413ee90 2024-03-29T11:26:29.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.378+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2003546677 0 0) 0x7f81740702c0 con 0x7f81740a9ac0 2024-03-29T11:26:29.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.378+0000 7f8169ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:29.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.378+0000 7f8169ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:29.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.378+0000 7f8169ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:29.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.378+0000 7f8169ffb640 10 monclient(hunting): my global_id is 4472 2024-03-29T11:26:29.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.379+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8174071d00 con 0x7f81740a9ac0 2024-03-29T11:26:29.380 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.379+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f8164001c00 con 0x7f81740b4100 2024-03-29T11:26:29.380 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.379+0000 7f8169ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:29.380 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.379+0000 7f8169ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:29.380 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.379+0000 7f8169ffb640 10 monclient(hunting): dump: 2024-03-29T11:26:29.380 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:29.380 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:29.380 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:29.380 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:29.380 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:29.380 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:29.380 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:29.380 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:29.380 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:29.380 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:29.380 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.379+0000 7f8169ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:29.381 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.379+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2354221603 0 0) 0x7f8158004eb0 con 0x7f817413ee90 2024-03-29T11:26:29.381 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.379+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81740702c0 con 0x7f817413ee90 2024-03-29T11:26:29.381 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.379+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 246573712 0 0) 0x7f8174071d00 con 0x7f81740a9ac0 2024-03-29T11:26:29.381 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.379+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8158004eb0 con 0x7f81740a9ac0 2024-03-29T11:26:29.381 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.379+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f816c001bf0 con 0x7f817413ee90 2024-03-29T11:26:29.381 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.379+0000 7f8169ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:29.381 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.379+0000 7f8169ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:29.381 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.379+0000 7f8169ffb640 10 monclient(hunting): dump: 2024-03-29T11:26:29.381 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:29.381 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:29.381 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:29.381 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:29.381 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:29.381 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:29.381 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:29.382 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:29.382 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:29.382 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:29.382 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.379+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f81700018b0 con 0x7f81740a9ac0 2024-03-29T11:26:29.382 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.379+0000 7f8169ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:29.382 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.379+0000 7f8169ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:29.382 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.379+0000 7f8169ffb640 10 monclient(hunting): dump: 2024-03-29T11:26:29.382 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:29.382 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:29.382 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:29.382 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:29.382 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:29.382 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:29.382 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:29.383 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:29.383 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:29.383 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:29.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.379+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 578028604 0 0) 0x7f8158004eb0 con 0x7f81740a9ac0 2024-03-29T11:26:29.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.379+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 >> v1:172.21.15.196:6789/0 conn(0x7f81740b4100 legacy=0x7f8174134730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:29.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.379+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 >> v1:172.21.15.86:6789/0 conn(0x7f817413ee90 legacy=0x7f81741e7290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:29.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.379+0000 7f8169ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:26:29.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.379+0000 7f8169ffb640 1 monclient: found mon.c 2024-03-29T11:26:29.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.379+0000 7f8169ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:29.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.379+0000 7f8169ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:26:29.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.379+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f817406eeb0 con 0x7f81740a9ac0 2024-03-29T11:26:29.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.379+0000 7f8169ffb640 10 monclient: _finish_auth 0 2024-03-29T11:26:29.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.379+0000 7f8169ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:26:29.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.379+0000 7f8169ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:29.384 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.380+0000 7f817ba01640 5 monclient: authenticate success, global_id 4472 2024-03-29T11:26:29.384 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.380+0000 7f817ba01640 10 monclient: _renew_subs 2024-03-29T11:26:29.384 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.380+0000 7f817ba01640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:26:29.384 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.380+0000 7f817ba01640 1 -- 172.21.15.86:0/3508716901 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f817406f8d0 con 0x7f81740a9ac0 2024-03-29T11:26:29.384 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.380+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8170001cc0 con 0x7f81740a9ac0 2024-03-29T11:26:29.384 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.380+0000 7f8169ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:29.384 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.380+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f8170004980 con 0x7f81740a9ac0 2024-03-29T11:26:29.384 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.380+0000 7f8169ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:29.384 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.380+0000 7f8169ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:29.384 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.380+0000 7f8169ffb640 10 monclient: dump: 2024-03-29T11:26:29.384 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:29.384 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:29.384 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:29.384 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:29.385 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:29.385 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:29.385 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:29.385 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:29.385 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:29.385 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:29.385 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.380+0000 7f817ba01640 10 client.4472.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:29.385 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.380+0000 7f817ba01640 10 monclient: _renew_subs 2024-03-29T11:26:29.385 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.380+0000 7f817ba01640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:26:29.385 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.380+0000 7f817ba01640 1 -- 172.21.15.86:0/3508716901 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81740709c0 con 0x7f81740a9ac0 2024-03-29T11:26:29.385 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.380+0000 7f817ba01640 1 librados: init done 2024-03-29T11:26:29.385 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.381+0000 7f817979d640 1 -- 172.21.15.86:0/3508716901 >> v1:172.21.15.86:6790/0 conn(0x7f81740a9ac0 legacy=0x7f817406e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-03-29T11:26:29.385 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.381+0000 7f817979d640 1 -- 172.21.15.86:0/3508716901 >> v1:172.21.15.86:6790/0 conn(0x7f81740a9ac0 legacy=0x7f817406e7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-29T11:26:29.385 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.381+0000 7f817979d640 1 --1- 172.21.15.86:0/3508716901 >> v1:172.21.15.86:6790/0 conn(0x7f81740a9ac0 0x7f817406e7a0 :-1 s=OPENED pgs=569729 cs=1 l=1).handle_message read tag failed 2024-03-29T11:26:29.385 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.381+0000 7f817979d640 1 --1- 172.21.15.86:0/3508716901 >> v1:172.21.15.86:6790/0 conn(0x7f81740a9ac0 0x7f817406e7a0 :-1 s=OPENED pgs=569729 cs=1 l=1).fault on lossy channel, failing 2024-03-29T11:26:29.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.381+0000 7f8169ffb640 10 monclient: ms_handle_reset current mon v1:172.21.15.86:6790/0 2024-03-29T11:26:29.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.381+0000 7f8169ffb640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:29.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.381+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 >> v1:172.21.15.86:6790/0 conn(0x7f81740a9ac0 legacy=0x7f817406e7a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-29T11:26:29.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.381+0000 7f8169ffb640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:29.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.381+0000 7f816affd640 4 set_mon_vals no callback set 2024-03-29T11:26:29.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.382+0000 7f817ba01640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:29.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.382+0000 7f8169ffb640 10 monclient(hunting): picked mon.b con 0x7f81740b4100 addr v1:172.21.15.196:6789/0 2024-03-29T11:26:29.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.382+0000 7f8169ffb640 10 monclient(hunting): picked mon.a con 0x7f817413ee90 addr v1:172.21.15.86:6789/0 2024-03-29T11:26:29.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.382+0000 7f8169ffb640 10 monclient(hunting): picked mon.c con 0x7f815800e480 addr v1:172.21.15.86:6790/0 2024-03-29T11:26:29.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.382+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8158004eb0 con 0x7f817413ee90 2024-03-29T11:26:29.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.382+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f81740702c0 con 0x7f81740b4100 2024-03-29T11:26:29.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.382+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 1) v1 -- 0x7f8174071d00 con 0x7f815800e480 2024-03-29T11:26:29.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.382+0000 7f8169ffb640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:29.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.382+0000 7f817ba01640 10 monclient(hunting): _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:29.387 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.386+0000 7f8169ffb640 10 client.4472.objecter ms_handle_connect 0x7f81740b4100 2024-03-29T11:26:29.387 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.386+0000 7f8169ffb640 10 client.4472.objecter resend_mon_ops 2024-03-29T11:26:29.387 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.386+0000 7f8169ffb640 10 client.4472.objecter ms_handle_connect 0x7f815800e480 2024-03-29T11:26:29.387 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.386+0000 7f8169ffb640 10 client.4472.objecter resend_mon_ops 2024-03-29T11:26:29.387 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.386+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3021010089 0 0) 0x7f81740702c0 con 0x7f81740b4100 2024-03-29T11:26:29.387 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.386+0000 7f8169ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:29.387 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.386+0000 7f8169ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:29.387 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.386+0000 7f8169ffb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T11:26:29.387 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.386+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 --> v1:172.21.15.196:6789/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81580131b0 con 0x7f81740b4100 2024-03-29T11:26:29.387 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.386+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 102935278 0 0) 0x7f8174071d00 con 0x7f815800e480 2024-03-29T11:26:29.387 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.386+0000 7f8169ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:29.387 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.386+0000 7f8169ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:29.387 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.386+0000 7f8169ffb640 10 monclient(hunting): _init_auth already have auth, reseting 2024-03-29T11:26:29.387 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.386+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 --> v1:172.21.15.86:6790/0 -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81740702c0 con 0x7f815800e480 2024-03-29T11:26:29.387 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.387+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (unknown 955028572 0 0) 0x7f81740702c0 con 0x7f815800e480 2024-03-29T11:26:29.388 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.387+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 >> v1:172.21.15.196:6789/0 conn(0x7f81740b4100 legacy=0x7f81580092f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:29.388 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.387+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 >> v1:172.21.15.86:6789/0 conn(0x7f817413ee90 legacy=0x7f8158009a00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:26:29.388 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.387+0000 7f8169ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:26:29.388 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.387+0000 7f8169ffb640 1 monclient: found mon.c 2024-03-29T11:26:29.388 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.387+0000 7f8169ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:29.388 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.387+0000 7f8169ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:26:29.388 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.387+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,mgrmap=0+,monmap=2+,osdmap=0}) v3 -- 0x7f8158015410 con 0x7f815800e480 2024-03-29T11:26:29.388 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.387+0000 7f8169ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:26:29.388 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.387+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8154005080 con 0x7f815800e480 2024-03-29T11:26:29.388 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.387+0000 7f8169ffb640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:29.388 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.387+0000 7f8169ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:26:29.388 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.387+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8158004eb0 con 0x7f815800e480 2024-03-29T11:26:29.388 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.387+0000 7f8169ffb640 10 monclient: _finish_auth 0 2024-03-29T11:26:29.388 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.387+0000 7f8169ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:26:29.388 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.387+0000 7f8169ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:29.389 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.387+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f816c003b30 con 0x7f815800e480 2024-03-29T11:26:29.389 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.387+0000 7f8169ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:29.389 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.387+0000 7f8169ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:29.389 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.387+0000 7f8169ffb640 10 monclient: dump: 2024-03-29T11:26:29.389 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:29.389 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:29.389 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:29.389 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:29.389 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:29.389 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:29.389 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:29.389 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:29.389 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:29.389 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:29.390 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.387+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f816c003c90 con 0x7f815800e480 2024-03-29T11:26:29.390 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.387+0000 7f8169ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:29.390 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.387+0000 7f816b7fe640 4 set_mon_vals no callback set 2024-03-29T11:26:29.390 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.389+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f816c019c70 con 0x7f815800e480 2024-03-29T11:26:29.390 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.390+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 <== mon.2 v1:172.21.15.86:6790/0 6 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f816c083aa0 con 0x7f815800e480 2024-03-29T11:26:29.390 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.390+0000 7f8169ffb640 10 client.4472.objecter ms_dispatch 0x7f8174000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:29.390 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.390+0000 7f8169ffb640 3 client.4472.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:29.390 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.390+0000 7f8169ffb640 3 client.4472.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:29.390 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.390+0000 7f8169ffb640 20 client.4472.objecter dump_active .. 0 homeless 2024-03-29T11:26:29.390 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.391+0000 7f817979d640 10 monclient: get_auth_request con 0x7f81740a9ac0 auth_method 0 2024-03-29T11:26:29.390 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.391+0000 7f8169ffb640 10 client.4472.objecter ms_handle_connect 0x7f81740a9ac0 2024-03-29T11:26:29.391 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.397+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f816c04f380 con 0x7f815800e480 2024-03-29T11:26:29.391 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.397+0000 7f8169ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:29.391 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.397+0000 7f8169ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:29.391 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.404+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 <== mon.2 v1:172.21.15.86:6790/0 8 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f816c0512b0 con 0x7f815800e480 2024-03-29T11:26:29.391 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.404+0000 7f8169ffb640 10 monclient: handle_mon_command_ack 1 not found 2024-03-29T11:26:29.708 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.722+0000 7f817ba01640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:29.709 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.722+0000 7f817ba01640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:29.709 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.722+0000 7f817ba01640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:26:29.709 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.722+0000 7f817ba01640 1 -- 172.21.15.86:0/3508716901 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8154004130 con 0x7f815800e480 2024-03-29T11:26:29.711 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.725+0000 7f8169ffb640 1 -- 172.21.15.86:0/3508716901 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f816c052ad0 con 0x7f815800e480 2024-03-29T11:26:29.711 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.725+0000 7f8169ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:29.711 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.725+0000 7f8169ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:29.711 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:26:29.711 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:29.714 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.728+0000 7f817ba01640 10 librados: watch_flush enter 2024-03-29T11:26:29.714 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.728+0000 7f817ba01640 10 librados: watch_flush exit 2024-03-29T11:26:29.714 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.728+0000 7f817ba01640 20 client.4472.objecter shutdown clearing up homeless session... 2024-03-29T11:26:29.715 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.728+0000 7f817ba01640 10 client.4472.objecter successfully canceled tick 2024-03-29T11:26:29.715 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.728+0000 7f817ba01640 1 -- 172.21.15.86:0/3508716901 >> v1:172.21.15.196:6816/588110318 conn(0x7f81740a9ac0 legacy=0x7f815807b960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:29.715 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.728+0000 7f817ba01640 10 monclient: shutdown 2024-03-29T11:26:29.715 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.728+0000 7f817ba01640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:29.716 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.728+0000 7f817ba01640 1 -- 172.21.15.86:0/3508716901 >> v1:172.21.15.86:6790/0 conn(0x7f815800e480 legacy=0x7f8158010870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:29.716 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.729+0000 7f817ba01640 1 -- 172.21.15.86:0/3508716901 shutdown_connections 2024-03-29T11:26:29.716 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.729+0000 7f817ba01640 1 -- 172.21.15.86:0/3508716901 >> 172.21.15.86:0/3508716901 conn(0x7f81740b35e0 msgr2=0x7f81740b07c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:29.716 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.729+0000 7f817ba01640 1 -- 172.21.15.86:0/3508716901 shutdown_connections 2024-03-29T11:26:29.716 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.729+0000 7f817ba01640 1 -- 172.21.15.86:0/3508716901 wait complete. 2024-03-29T11:26:29.716 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:29.729+0000 7f817ba01640 1 librados: shutdown 2024-03-29T11:26:29.732 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:36.539 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:36.687 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.700+0000 7f1146df6640 10 monclient: get_monmap_and_config 2024-03-29T11:26:36.687 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.700+0000 7f1146df6640 10 monclient: build_initial_monmap 2024-03-29T11:26:36.687 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.700+0000 7f1146df6640 10 monclient: monmap: 2024-03-29T11:26:36.687 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:26:36.687 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:36.687 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:26:36.701224+0000 2024-03-29T11:26:36.688 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:26:36.701224+0000 2024-03-29T11:26:36.688 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:36.688 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:36.688 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:26:36.688 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:26:36.688 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:26:36.688 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:36.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.701+0000 7f1146df6640 1 Processor -- start 2024-03-29T11:26:36.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.701+0000 7f1146df6640 1 -- start start 2024-03-29T11:26:36.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.701+0000 7f1146df6640 10 monclient: init 2024-03-29T11:26:36.691 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.702+0000 7f1146df6640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:36.691 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.702+0000 7f1146df6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:26:36.691 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.702+0000 7f1146df6640 10 monclient(hunting): picked mon.noname-c con 0x7f11400b0770 addr v1:172.21.15.196:6789/0 2024-03-29T11:26:36.691 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.702+0000 7f1146df6640 10 monclient(hunting): picked mon.noname-a con 0x7f114014a700 addr v1:172.21.15.86:6789/0 2024-03-29T11:26:36.691 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.702+0000 7f1146df6640 10 monclient(hunting): picked mon.noname-b con 0x7f11400ac5a0 addr v1:172.21.15.86:6790/0 2024-03-29T11:26:36.691 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.702+0000 7f1146df6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11400aba70 con 0x7f114014a700 2024-03-29T11:26:36.691 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.702+0000 7f1146df6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11400aecb0 con 0x7f11400b0770 2024-03-29T11:26:36.691 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.702+0000 7f1146df6640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11400af150 con 0x7f11400ac5a0 2024-03-29T11:26:36.691 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.702+0000 7f1146df6640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:36.692 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.702+0000 7f1146df6640 10 monclient(hunting): authenticate will time out at 1221.020499s 2024-03-29T11:26:36.692 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.705+0000 7f1145393640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f11400ac5a0 0x7f1140151f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:43784/0 (socket says 172.21.15.86:43784) 2024-03-29T11:26:36.692 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.705+0000 7f1145393640 1 -- 172.21.15.86:0/303313935 learned_addr learned my addr 172.21.15.86:0/303313935 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:26:36.693 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.705+0000 7f113f7fe640 1 -- 172.21.15.86:0/303313935 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 916277192 0 0) 0x7f11400aba70 con 0x7f114014a700 2024-03-29T11:26:36.693 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.705+0000 7f113f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:36.693 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.705+0000 7f113f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:36.693 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.705+0000 7f113f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:36.693 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.705+0000 7f113f7fe640 10 monclient(hunting): my global_id is 4425 2024-03-29T11:26:36.693 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.705+0000 7f113f7fe640 1 -- 172.21.15.86:0/303313935 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f112c003520 con 0x7f114014a700 2024-03-29T11:26:36.693 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.706+0000 7f113f7fe640 1 -- 172.21.15.86:0/303313935 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 574801270 0 0) 0x7f11400af150 con 0x7f11400ac5a0 2024-03-29T11:26:36.693 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.706+0000 7f113f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:26:36.693 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.706+0000 7f113f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:36.693 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.706+0000 7f113f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:36.693 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.706+0000 7f113f7fe640 10 monclient(hunting): my global_id is 4475 2024-03-29T11:26:36.693 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.706+0000 7f113f7fe640 1 -- 172.21.15.86:0/303313935 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f11400aba70 con 0x7f11400ac5a0 2024-03-29T11:26:36.693 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.706+0000 7f113f7fe640 1 -- 172.21.15.86:0/303313935 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3054190906 0 0) 0x7f11400aba70 con 0x7f11400ac5a0 2024-03-29T11:26:36.694 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.706+0000 7f113f7fe640 1 -- 172.21.15.86:0/303313935 >> v1:172.21.15.196:6789/0 conn(0x7f11400b0770 legacy=0x7f1140149ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:26:36.696 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.709+0000 7f113f7fe640 1 -- 172.21.15.86:0/303313935 >> v1:172.21.15.86:6789/0 conn(0x7f114014a700 legacy=0x7f11400abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:36.696 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.709+0000 7f113f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:26:36.696 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.709+0000 7f113f7fe640 1 monclient: found mon.noname-b 2024-03-29T11:26:36.696 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.709+0000 7f113f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:36.696 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.709+0000 7f113f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:26:36.696 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.709+0000 7f113f7fe640 1 -- 172.21.15.86:0/303313935 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11400ac9d0 con 0x7f11400ac5a0 2024-03-29T11:26:36.697 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.709+0000 7f113f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:26:36.698 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.709+0000 7f113f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:26:36.698 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.709+0000 7f113f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:36.698 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.709+0000 7f1146df6640 5 monclient: authenticate success, global_id 4475 2024-03-29T11:26:36.699 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.709+0000 7f1146df6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:36.699 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.709+0000 7f113f7fe640 1 -- 172.21.15.86:0/303313935 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f1128001670 con 0x7f11400ac5a0 2024-03-29T11:26:36.699 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.709+0000 7f113f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:36.699 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.709+0000 7f113f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:26:36.699 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.709+0000 7f113f7fe640 10 monclient: dump: 2024-03-29T11:26:36.699 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:36.699 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:36.699 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:36.699 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:36.700 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:36.700 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:36.700 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:36.700 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:36.700 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:36.700 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:36.700 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.709+0000 7f1146df6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:36.700 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.709+0000 7f113f7fe640 1 -- 172.21.15.86:0/303313935 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f11280017d0 con 0x7f11400ac5a0 2024-03-29T11:26:36.700 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.710+0000 7f113f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:36.701 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.710+0000 7f113f7fe640 1 -- 172.21.15.86:0/303313935 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f1128001ba0 con 0x7f11400ac5a0 2024-03-29T11:26:36.701 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.710+0000 7f113f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:36.701 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.710+0000 7f113f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:36.701 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.710+0000 7f113f7fe640 10 monclient: dump: 2024-03-29T11:26:36.701 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:36.701 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:36.701 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:36.701 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:36.701 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:36.702 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:36.702 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:36.702 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:36.702 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:36.702 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:36.702 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.710+0000 7f1146df6640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:36.702 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.710+0000 7f1146df6640 4 set_mon_vals no callback set 2024-03-29T11:26:36.714 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.711+0000 7f1146df6640 10 monclient: shutdown 2024-03-29T11:26:36.714 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.711+0000 7f1146df6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:36.714 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.711+0000 7f1146df6640 1 -- 172.21.15.86:0/303313935 >> v1:172.21.15.86:6790/0 conn(0x7f11400ac5a0 legacy=0x7f1140151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:36.715 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.713+0000 7f1146df6640 1 -- 172.21.15.86:0/303313935 shutdown_connections 2024-03-29T11:26:36.715 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.713+0000 7f1146df6640 1 -- 172.21.15.86:0/303313935 >> 172.21.15.86:0/303313935 conn(0x7f114006b660 msgr2=0x7f114006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:36.715 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.713+0000 7f1146df6640 1 -- 172.21.15.86:0/303313935 shutdown_connections 2024-03-29T11:26:36.715 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.713+0000 7f1146df6640 1 -- 172.21.15.86:0/303313935 wait complete. 2024-03-29T11:26:36.715 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.714+0000 7f1146df6640 10 monclient: build_initial_monmap 2024-03-29T11:26:36.715 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.714+0000 7f1146df6640 10 monclient: monmap: 2024-03-29T11:26:36.715 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:26:36.715 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:36.715 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:26:36.715 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:26:36.715 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:36.715 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:36.716 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:26:36.716 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:26:36.716 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:26:36.716 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:36.716 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.714+0000 7f1146df6640 1 librados: starting msgr at 2024-03-29T11:26:36.716 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.715+0000 7f1146df6640 1 librados: starting objecter 2024-03-29T11:26:36.716 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.715+0000 7f1146df6640 1 Processor -- start 2024-03-29T11:26:36.716 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.715+0000 7f1146df6640 1 -- start start 2024-03-29T11:26:36.716 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.715+0000 7f1146df6640 1 librados: setting wanted keys 2024-03-29T11:26:36.716 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.715+0000 7f1146df6640 1 librados: calling monclient init 2024-03-29T11:26:36.716 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.715+0000 7f1146df6640 10 monclient: init 2024-03-29T11:26:36.716 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.716+0000 7f1146df6640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:36.716 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.716+0000 7f1146df6640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:26:36.716 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.716+0000 7f1146df6640 10 monclient(hunting): picked mon.noname-c con 0x7f11400ac5a0 addr v1:172.21.15.86:6790/0 2024-03-29T11:26:36.716 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.716+0000 7f1146df6640 10 monclient(hunting): picked mon.noname-b con 0x7f11400b0770 addr v1:172.21.15.196:6789/0 2024-03-29T11:26:36.717 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.716+0000 7f1146df6640 10 monclient(hunting): picked mon.noname-a con 0x7f114014a700 addr v1:172.21.15.86:6789/0 2024-03-29T11:26:36.717 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.716+0000 7f1146df6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11401239d0 con 0x7f114014a700 2024-03-29T11:26:36.717 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.716+0000 7f1146df6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1140128870 con 0x7f11400b0770 2024-03-29T11:26:36.717 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.717+0000 7f1146df6640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11401267c0 con 0x7f11400ac5a0 2024-03-29T11:26:36.717 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.717+0000 7f1146df6640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:36.717 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.717+0000 7f1146df6640 10 monclient(hunting): authenticate will time out at 1221.035082s 2024-03-29T11:26:36.717 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.718+0000 7f1144b92640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f11400ac5a0 0x7f1140145180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:44178/0 (socket says 172.21.15.86:44178) 2024-03-29T11:26:36.717 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.718+0000 7f1144b92640 1 -- 172.21.15.86:0/1726344403 learned_addr learned my addr 172.21.15.86:0/1726344403 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:26:36.717 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.719+0000 7f113d7fa640 10 client.?.objecter ms_handle_connect 0x7f11400ac5a0 2024-03-29T11:26:36.717 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.719+0000 7f113d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:36.717 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.719+0000 7f113d7fa640 1 -- 172.21.15.86:0/1726344403 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 466693568 0 0) 0x7f11401267c0 con 0x7f11400ac5a0 2024-03-29T11:26:36.717 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.719+0000 7f113d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:26:36.717 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.719+0000 7f113d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:36.717 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.719+0000 7f113d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:36.718 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.719+0000 7f113d7fa640 10 monclient(hunting): my global_id is 4478 2024-03-29T11:26:36.718 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.719+0000 7f113d7fa640 1 -- 172.21.15.86:0/1726344403 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1120004eb0 con 0x7f11400ac5a0 2024-03-29T11:26:36.718 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.719+0000 7f113d7fa640 1 -- 172.21.15.86:0/1726344403 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2171671756 0 0) 0x7f1120004eb0 con 0x7f11400ac5a0 2024-03-29T11:26:36.718 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.719+0000 7f113d7fa640 1 -- 172.21.15.86:0/1726344403 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f11401267c0 con 0x7f11400ac5a0 2024-03-29T11:26:36.718 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.719+0000 7f113d7fa640 1 -- 172.21.15.86:0/1726344403 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f1130001870 con 0x7f11400ac5a0 2024-03-29T11:26:36.718 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.719+0000 7f113d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:36.718 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.719+0000 7f113d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:36.719 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.719+0000 7f113d7fa640 10 monclient(hunting): dump: 2024-03-29T11:26:36.719 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:36.719 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:36.719 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:36.719 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:36.719 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:36.719 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:36.719 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:36.719 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:36.720 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:36.720 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:36.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.719+0000 7f113d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:36.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.719+0000 7f113d7fa640 1 -- 172.21.15.86:0/1726344403 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 224005732 0 0) 0x7f11401267c0 con 0x7f11400ac5a0 2024-03-29T11:26:36.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.719+0000 7f113d7fa640 1 -- 172.21.15.86:0/1726344403 >> v1:172.21.15.196:6789/0 conn(0x7f11400b0770 legacy=0x7f1140126ce0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:26:36.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.720+0000 7f113d7fa640 1 -- 172.21.15.86:0/1726344403 >> v1:172.21.15.86:6789/0 conn(0x7f114014a700 legacy=0x7f11401e7200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:26:36.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.722+0000 7f113d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:26:36.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.722+0000 7f113d7fa640 1 monclient: found mon.c 2024-03-29T11:26:36.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.722+0000 7f113d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:36.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.722+0000 7f113d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:26:36.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.722+0000 7f113d7fa640 1 -- 172.21.15.86:0/1726344403 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11401dfb10 con 0x7f11400ac5a0 2024-03-29T11:26:36.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.722+0000 7f113d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:26:36.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.722+0000 7f113d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:26:36.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.722+0000 7f113d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:36.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.722+0000 7f1146df6640 5 monclient: authenticate success, global_id 4478 2024-03-29T11:26:36.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.722+0000 7f113d7fa640 1 -- 172.21.15.86:0/1726344403 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f1130001ad0 con 0x7f11400ac5a0 2024-03-29T11:26:36.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.722+0000 7f1146df6640 10 monclient: _renew_subs 2024-03-29T11:26:36.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.722+0000 7f1146df6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:26:36.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.722+0000 7f1146df6640 1 -- 172.21.15.86:0/1726344403 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11401dfd60 con 0x7f11400ac5a0 2024-03-29T11:26:36.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.723+0000 7f1146df6640 10 client.4478.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:36.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.723+0000 7f113d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:36.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.723+0000 7f113d7fa640 1 -- 172.21.15.86:0/1726344403 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f1130004b60 con 0x7f11400ac5a0 2024-03-29T11:26:36.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.723+0000 7f113d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:36.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.723+0000 7f113effd640 4 set_mon_vals no callback set 2024-03-29T11:26:36.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.723+0000 7f113d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:36.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.723+0000 7f113d7fa640 10 monclient: dump: 2024-03-29T11:26:36.721 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:36.722 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:36.722 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:36.722 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:36.722 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:36.722 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:36.722 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:36.722 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:36.722 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:36.722 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:36.722 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.723+0000 7f1146df6640 10 monclient: _renew_subs 2024-03-29T11:26:36.722 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.723+0000 7f1146df6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:26:36.722 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.723+0000 7f1146df6640 1 -- 172.21.15.86:0/1726344403 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f11401263f0 con 0x7f11400ac5a0 2024-03-29T11:26:36.722 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.723+0000 7f1146df6640 1 librados: init done 2024-03-29T11:26:36.722 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.724+0000 7f113d7fa640 1 -- 172.21.15.86:0/1726344403 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f113001ab30 con 0x7f11400ac5a0 2024-03-29T11:26:36.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.725+0000 7f1146df6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:36.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.725+0000 7f1146df6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:36.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.725+0000 7f1146df6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:26:36.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.725+0000 7f1146df6640 1 -- 172.21.15.86:0/1726344403 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1140126a00 con 0x7f11400ac5a0 2024-03-29T11:26:36.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.725+0000 7f113d7fa640 1 -- 172.21.15.86:0/1726344403 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f1130084bf0 con 0x7f11400ac5a0 2024-03-29T11:26:36.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.725+0000 7f113d7fa640 10 client.4478.objecter ms_dispatch 0x7f1140000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:36.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.725+0000 7f113d7fa640 3 client.4478.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:36.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.725+0000 7f113d7fa640 3 client.4478.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:36.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.725+0000 7f113d7fa640 20 client.4478.objecter dump_active .. 0 homeless 2024-03-29T11:26:36.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.725+0000 7f113ffff640 10 monclient: get_auth_request con 0x7f112006dc10 auth_method 0 2024-03-29T11:26:36.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.726+0000 7f113d7fa640 10 client.4478.objecter ms_handle_connect 0x7f112006dc10 2024-03-29T11:26:36.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.735+0000 7f113d7fa640 1 -- 172.21.15.86:0/1726344403 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f1140126a00 con 0x7f11400ac5a0 2024-03-29T11:26:36.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.735+0000 7f113d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:36.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:36.735+0000 7f113d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:37.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:37.057+0000 7f1146df6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:37.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:37.058+0000 7f1146df6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:37.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:37.058+0000 7f1146df6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:26:37.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:37.058+0000 7f1146df6640 1 -- 172.21.15.86:0/1726344403 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1140126a00 con 0x7f11400ac5a0 2024-03-29T11:26:37.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:37.059+0000 7f113d7fa640 1 -- 172.21.15.86:0/1726344403 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f1140126a00 con 0x7f11400ac5a0 2024-03-29T11:26:37.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:37.059+0000 7f113d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:37.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:37.059+0000 7f113d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:37.046 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:26:37.047 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:37.049 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:37.063+0000 7f1146df6640 10 librados: watch_flush enter 2024-03-29T11:26:37.049 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:37.063+0000 7f1146df6640 10 librados: watch_flush exit 2024-03-29T11:26:37.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:37.063+0000 7f1146df6640 20 client.4478.objecter shutdown clearing up homeless session... 2024-03-29T11:26:37.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:37.063+0000 7f1146df6640 10 client.4478.objecter successfully canceled tick 2024-03-29T11:26:37.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:37.063+0000 7f1146df6640 1 -- 172.21.15.86:0/1726344403 >> v1:172.21.15.196:6816/588110318 conn(0x7f112006dc10 legacy=0x7f11200700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:37.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:37.063+0000 7f1146df6640 10 monclient: shutdown 2024-03-29T11:26:37.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:37.063+0000 7f1146df6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:37.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:37.063+0000 7f1146df6640 1 -- 172.21.15.86:0/1726344403 >> v1:172.21.15.86:6790/0 conn(0x7f11400ac5a0 legacy=0x7f1140145180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:37.051 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:37.064+0000 7f1146df6640 1 -- 172.21.15.86:0/1726344403 shutdown_connections 2024-03-29T11:26:37.051 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:37.064+0000 7f1146df6640 1 -- 172.21.15.86:0/1726344403 >> 172.21.15.86:0/1726344403 conn(0x7f114006b660 msgr2=0x7f114006ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:37.051 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:37.064+0000 7f1146df6640 1 -- 172.21.15.86:0/1726344403 shutdown_connections 2024-03-29T11:26:37.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:37.064+0000 7f1146df6640 1 -- 172.21.15.86:0/1726344403 wait complete. 2024-03-29T11:26:37.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:37.064+0000 7f1146df6640 1 librados: shutdown 2024-03-29T11:26:37.067 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:43.875 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:44.024 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.037+0000 7fc9fc6d6640 10 monclient: get_monmap_and_config 2024-03-29T11:26:44.024 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.037+0000 7fc9fc6d6640 10 monclient: build_initial_monmap 2024-03-29T11:26:44.024 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.037+0000 7fc9fc6d6640 10 monclient: monmap: 2024-03-29T11:26:44.024 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:26:44.024 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:44.024 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:26:44.038561+0000 2024-03-29T11:26:44.024 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:26:44.038561+0000 2024-03-29T11:26:44.024 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:44.025 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:44.025 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:26:44.025 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:26:44.025 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:26:44.025 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:44.026 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.038+0000 7fc9fc6d6640 1 Processor -- start 2024-03-29T11:26:44.026 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.038+0000 7fc9fc6d6640 1 -- start start 2024-03-29T11:26:44.026 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.038+0000 7fc9fc6d6640 10 monclient: init 2024-03-29T11:26:44.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.039+0000 7fc9fc6d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:44.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.039+0000 7fc9fc6d6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:44.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.039+0000 7fc9fc6d6640 10 monclient(hunting): picked mon.noname-b con 0x7fc9f41457f0 addr v1:172.21.15.86:6790/0 2024-03-29T11:26:44.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.039+0000 7fc9fc6d6640 10 monclient(hunting): picked mon.noname-a con 0x7fc9f413f650 addr v1:172.21.15.86:6789/0 2024-03-29T11:26:44.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.039+0000 7fc9fc6d6640 10 monclient(hunting): picked mon.noname-c con 0x7fc9f40a9d70 addr v1:172.21.15.196:6789/0 2024-03-29T11:26:44.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.039+0000 7fc9fc6d6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9f40aaef0 con 0x7fc9f413f650 2024-03-29T11:26:44.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.039+0000 7fc9fc6d6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9f4145bd0 con 0x7fc9f40a9d70 2024-03-29T11:26:44.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.039+0000 7fc9fc6d6640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9f4145eb0 con 0x7fc9f41457f0 2024-03-29T11:26:44.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.039+0000 7fc9fc6d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:44.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.039+0000 7fc9fc6d6640 10 monclient(hunting): authenticate will time out at 1228.357607s 2024-03-29T11:26:44.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.043+0000 7fc9fa472640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7fc9f41457f0 0x7fc9f413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:33256/0 (socket says 172.21.15.86:33256) 2024-03-29T11:26:44.030 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.043+0000 7fc9fa472640 1 -- 172.21.15.86:0/1400771724 learned_addr learned my addr 172.21.15.86:0/1400771724 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:26:44.034 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.043+0000 7fc9f9470640 1 -- 172.21.15.86:0/1400771724 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4104931806 0 0) 0x7fc9f40aaef0 con 0x7fc9f413f650 2024-03-29T11:26:44.034 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.044+0000 7fc9f9470640 10 monclient(hunting): _negotiate 2024-03-29T11:26:44.034 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.044+0000 7fc9f9470640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:44.035 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.044+0000 7fc9f9470640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:44.035 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.044+0000 7fc9f9470640 10 monclient(hunting): my global_id is 4428 2024-03-29T11:26:44.035 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.044+0000 7fc9f9470640 1 -- 172.21.15.86:0/1400771724 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9e0003520 con 0x7fc9f413f650 2024-03-29T11:26:44.035 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.044+0000 7fc9f9470640 1 -- 172.21.15.86:0/1400771724 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 111679649 0 0) 0x7fc9f4145eb0 con 0x7fc9f41457f0 2024-03-29T11:26:44.035 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.044+0000 7fc9f9470640 10 monclient(hunting): _negotiate 2024-03-29T11:26:44.035 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.045+0000 7fc9f9470640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:44.035 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.045+0000 7fc9f9470640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:44.035 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.045+0000 7fc9f9470640 10 monclient(hunting): my global_id is 4481 2024-03-29T11:26:44.035 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.045+0000 7fc9f9470640 1 -- 172.21.15.86:0/1400771724 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9f40aaef0 con 0x7fc9f41457f0 2024-03-29T11:26:44.035 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.045+0000 7fc9f9470640 1 -- 172.21.15.86:0/1400771724 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1579849733 0 0) 0x7fc9e0003520 con 0x7fc9f413f650 2024-03-29T11:26:44.035 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.045+0000 7fc9f9470640 1 -- 172.21.15.86:0/1400771724 >> v1:172.21.15.86:6790/0 conn(0x7fc9f41457f0 legacy=0x7fc9f413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:44.035 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.045+0000 7fc9f9470640 1 -- 172.21.15.86:0/1400771724 >> v1:172.21.15.196:6789/0 conn(0x7fc9f40a9d70 legacy=0x7fc9f414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:44.035 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.045+0000 7fc9f9470640 10 monclient: _finish_hunting 0 2024-03-29T11:26:44.035 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.045+0000 7fc9f9470640 1 monclient: found mon.noname-a 2024-03-29T11:26:44.036 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.045+0000 7fc9f9470640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:44.036 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.045+0000 7fc9f9470640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.86:6789/0 2024-03-29T11:26:44.036 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.045+0000 7fc9f9470640 1 -- 172.21.15.86:0/1400771724 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f4146340 con 0x7fc9f413f650 2024-03-29T11:26:44.036 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.045+0000 7fc9f9470640 10 monclient: _finish_auth 0 2024-03-29T11:26:44.036 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.045+0000 7fc9f9470640 10 monclient: _check_auth_tickets 2024-03-29T11:26:44.036 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.045+0000 7fc9f9470640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:44.036 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.045+0000 7fc9f9470640 1 -- 172.21.15.86:0/1400771724 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc9f0001690 con 0x7fc9f413f650 2024-03-29T11:26:44.036 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.045+0000 7fc9f9470640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:44.036 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.046+0000 7fc9f9470640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:26:44.036 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.046+0000 7fc9f9470640 10 monclient: dump: 2024-03-29T11:26:44.036 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:44.036 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:44.036 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:44.036 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:44.036 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:44.037 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:44.037 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:44.037 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:44.037 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:44.037 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:44.037 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.046+0000 7fc9f9470640 1 -- 172.21.15.86:0/1400771724 <== mon.0 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9f00018b0 con 0x7fc9f413f650 2024-03-29T11:26:44.037 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.046+0000 7fc9f9470640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:44.037 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.046+0000 7fc9f9470640 1 -- 172.21.15.86:0/1400771724 <== mon.0 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc9f0001c40 con 0x7fc9f413f650 2024-03-29T11:26:44.037 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.046+0000 7fc9f9470640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:44.037 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.046+0000 7fc9f9470640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:44.037 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.046+0000 7fc9f9470640 10 monclient: dump: 2024-03-29T11:26:44.037 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:44.037 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:44.037 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:44.038 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:44.038 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:44.038 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:44.038 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:44.038 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:44.038 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:44.038 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:44.038 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.046+0000 7fc9fc6d6640 5 monclient: authenticate success, global_id 4428 2024-03-29T11:26:44.038 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.046+0000 7fc9fc6d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:44.038 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.046+0000 7fc9fc6d6640 4 set_mon_vals no callback set 2024-03-29T11:26:44.038 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.047+0000 7fc9fc6d6640 10 monclient: shutdown 2024-03-29T11:26:44.038 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.047+0000 7fc9fc6d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:44.039 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.047+0000 7fc9fc6d6640 1 -- 172.21.15.86:0/1400771724 >> v1:172.21.15.86:6789/0 conn(0x7fc9f413f650 legacy=0x7fc9f40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:44.039 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.047+0000 7fc9fc6d6640 1 -- 172.21.15.86:0/1400771724 shutdown_connections 2024-03-29T11:26:44.039 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.047+0000 7fc9fc6d6640 1 -- 172.21.15.86:0/1400771724 >> 172.21.15.86:0/1400771724 conn(0x7fc9f4066a00 msgr2=0x7fc9f4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:44.039 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.048+0000 7fc9fc6d6640 1 -- 172.21.15.86:0/1400771724 shutdown_connections 2024-03-29T11:26:44.043 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.048+0000 7fc9fc6d6640 1 -- 172.21.15.86:0/1400771724 wait complete. 2024-03-29T11:26:44.043 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.049+0000 7fc9fc6d6640 10 monclient: build_initial_monmap 2024-03-29T11:26:44.043 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.049+0000 7fc9fc6d6640 10 monclient: monmap: 2024-03-29T11:26:44.043 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:26:44.044 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:44.044 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:26:44.044 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:26:44.044 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:44.044 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:44.044 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:26:44.044 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:26:44.044 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:26:44.044 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:44.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.049+0000 7fc9fc6d6640 1 librados: starting msgr at 2024-03-29T11:26:44.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.049+0000 7fc9fc6d6640 1 librados: starting objecter 2024-03-29T11:26:44.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.049+0000 7fc9fc6d6640 1 Processor -- start 2024-03-29T11:26:44.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.050+0000 7fc9fc6d6640 1 -- start start 2024-03-29T11:26:44.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.050+0000 7fc9fc6d6640 1 librados: setting wanted keys 2024-03-29T11:26:44.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.050+0000 7fc9fc6d6640 1 librados: calling monclient init 2024-03-29T11:26:44.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.050+0000 7fc9fc6d6640 10 monclient: init 2024-03-29T11:26:44.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.051+0000 7fc9fc6d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:44.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.051+0000 7fc9fc6d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:26:44.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.051+0000 7fc9fc6d6640 10 monclient(hunting): picked mon.noname-a con 0x7fc9f40a9d70 addr v1:172.21.15.86:6789/0 2024-03-29T11:26:44.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.051+0000 7fc9fc6d6640 10 monclient(hunting): picked mon.noname-c con 0x7fc9f413f650 addr v1:172.21.15.86:6790/0 2024-03-29T11:26:44.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.051+0000 7fc9fc6d6640 10 monclient(hunting): picked mon.noname-b con 0x7fc9f41457f0 addr v1:172.21.15.196:6789/0 2024-03-29T11:26:44.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.051+0000 7fc9fc6d6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9f4141cf0 con 0x7fc9f40a9d70 2024-03-29T11:26:44.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.051+0000 7fc9fc6d6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9f4141fd0 con 0x7fc9f41457f0 2024-03-29T11:26:44.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.051+0000 7fc9fc6d6640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9f4140920 con 0x7fc9f413f650 2024-03-29T11:26:44.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.051+0000 7fc9fc6d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:44.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.051+0000 7fc9fc6d6640 10 monclient(hunting): authenticate will time out at 1228.369351s 2024-03-29T11:26:44.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.053+0000 7fc9f9c71640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7fc9f413f650 0x7fc9f4139f70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:58146/0 (socket says 172.21.15.86:58146) 2024-03-29T11:26:44.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.053+0000 7fc9f9c71640 1 -- 172.21.15.86:0/241891431 learned_addr learned my addr 172.21.15.86:0/241891431 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:26:44.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.053+0000 7fc9e6ffd640 10 client.?.objecter ms_handle_connect 0x7fc9f413f650 2024-03-29T11:26:44.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.053+0000 7fc9e6ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:44.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.053+0000 7fc9e6ffd640 1 -- 172.21.15.86:0/241891431 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2227209037 0 0) 0x7fc9f4140920 con 0x7fc9f413f650 2024-03-29T11:26:44.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.053+0000 7fc9e6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:44.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.053+0000 7fc9e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:44.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.053+0000 7fc9e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:44.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.054+0000 7fc9e6ffd640 10 monclient(hunting): my global_id is 4484 2024-03-29T11:26:44.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.054+0000 7fc9e6ffd640 1 -- 172.21.15.86:0/241891431 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9d8004eb0 con 0x7fc9f413f650 2024-03-29T11:26:44.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.054+0000 7fc9e6ffd640 1 -- 172.21.15.86:0/241891431 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2587195020 0 0) 0x7fc9d8004eb0 con 0x7fc9f413f650 2024-03-29T11:26:44.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.054+0000 7fc9e6ffd640 1 -- 172.21.15.86:0/241891431 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc9f4140920 con 0x7fc9f413f650 2024-03-29T11:26:44.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.055+0000 7fc9e6ffd640 1 -- 172.21.15.86:0/241891431 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc9f0001690 con 0x7fc9f413f650 2024-03-29T11:26:44.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.055+0000 7fc9e6ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:44.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.055+0000 7fc9e6ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:44.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.055+0000 7fc9e6ffd640 10 monclient(hunting): dump: 2024-03-29T11:26:44.047 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:44.047 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:44.047 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:44.047 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:44.047 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:44.047 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:44.047 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:44.047 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:44.047 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:44.047 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:44.047 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.055+0000 7fc9e6ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:44.047 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.055+0000 7fc9e6ffd640 1 -- 172.21.15.86:0/241891431 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3839493404 0 0) 0x7fc9f4140920 con 0x7fc9f413f650 2024-03-29T11:26:44.047 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.055+0000 7fc9e6ffd640 1 -- 172.21.15.86:0/241891431 >> v1:172.21.15.196:6789/0 conn(0x7fc9f41457f0 legacy=0x7fc9f41e66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:44.047 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.055+0000 7fc9e6ffd640 1 -- 172.21.15.86:0/241891431 >> v1:172.21.15.86:6789/0 conn(0x7fc9f40a9d70 legacy=0x7fc9f41423f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:44.047 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.055+0000 7fc9e6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:44.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.055+0000 7fc9e6ffd640 1 monclient: found mon.c 2024-03-29T11:26:44.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.055+0000 7fc9e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:44.049 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.055+0000 7fc9e6ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:26:44.049 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.055+0000 7fc9e6ffd640 1 -- 172.21.15.86:0/241891431 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9f41351c0 con 0x7fc9f413f650 2024-03-29T11:26:44.049 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.055+0000 7fc9e6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:26:44.049 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.055+0000 7fc9e6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:44.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.055+0000 7fc9e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:44.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.055+0000 7fc9fc6d6640 5 monclient: authenticate success, global_id 4484 2024-03-29T11:26:44.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.055+0000 7fc9fc6d6640 10 monclient: _renew_subs 2024-03-29T11:26:44.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.055+0000 7fc9fc6d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:26:44.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.055+0000 7fc9fc6d6640 1 -- 172.21.15.86:0/241891431 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9f4142b00 con 0x7fc9f413f650 2024-03-29T11:26:44.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.055+0000 7fc9fc6d6640 10 client.4484.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:44.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.055+0000 7fc9fc6d6640 10 monclient: _renew_subs 2024-03-29T11:26:44.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.056+0000 7fc9fc6d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:26:44.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.056+0000 7fc9fc6d6640 1 -- 172.21.15.86:0/241891431 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9f41cee00 con 0x7fc9f413f650 2024-03-29T11:26:44.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.056+0000 7fc9fc6d6640 1 librados: init done 2024-03-29T11:26:44.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.056+0000 7fc9e6ffd640 1 -- 172.21.15.86:0/241891431 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc9f00038a0 con 0x7fc9f413f650 2024-03-29T11:26:44.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.056+0000 7fc9e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:44.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.056+0000 7fc9e6ffd640 1 -- 172.21.15.86:0/241891431 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc9f0003cb0 con 0x7fc9f413f650 2024-03-29T11:26:44.051 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.056+0000 7fc9f8c6f640 4 set_mon_vals no callback set 2024-03-29T11:26:44.051 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.056+0000 7fc9e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:44.051 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.056+0000 7fc9e6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:44.051 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.056+0000 7fc9e6ffd640 10 monclient: dump: 2024-03-29T11:26:44.051 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:44.051 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:44.051 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:44.051 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:44.051 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:44.051 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:44.051 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:44.051 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:44.051 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:44.051 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:44.053 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.057+0000 7fc9e6ffd640 1 -- 172.21.15.86:0/241891431 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7fc9f0003e10 con 0x7fc9f413f650 2024-03-29T11:26:44.053 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.059+0000 7fc9e6ffd640 1 -- 172.21.15.86:0/241891431 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7fc9f0084be0 con 0x7fc9f413f650 2024-03-29T11:26:44.053 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.059+0000 7fc9e6ffd640 10 client.4484.objecter ms_dispatch 0x7fc9f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:44.053 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.059+0000 7fc9e6ffd640 3 client.4484.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:44.053 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.059+0000 7fc9e6ffd640 3 client.4484.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:44.053 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.059+0000 7fc9e6ffd640 20 client.4484.objecter dump_active .. 0 homeless 2024-03-29T11:26:44.053 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.060+0000 7fc9fc6d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:44.053 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.060+0000 7fc9fc6d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:44.053 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.060+0000 7fc9fc6d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:26:44.053 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.060+0000 7fc9fc6d6640 1 -- 172.21.15.86:0/241891431 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9dc005080 con 0x7fc9f413f650 2024-03-29T11:26:44.053 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.061+0000 7fc9fa472640 10 monclient: get_auth_request con 0x7fc9d806dc40 auth_method 0 2024-03-29T11:26:44.053 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.061+0000 7fc9e6ffd640 10 client.4484.objecter ms_handle_connect 0x7fc9d806dc40 2024-03-29T11:26:44.056 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.069+0000 7fc9e6ffd640 1 -- 172.21.15.86:0/241891431 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc9f00504c0 con 0x7fc9f413f650 2024-03-29T11:26:44.056 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.069+0000 7fc9e6ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:44.056 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.069+0000 7fc9e6ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:44.393 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.406+0000 7fc9fc6d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:44.393 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.406+0000 7fc9fc6d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:44.394 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.406+0000 7fc9fc6d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:26:44.394 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.406+0000 7fc9fc6d6640 1 -- 172.21.15.86:0/241891431 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc9dc005330 con 0x7fc9f413f650 2024-03-29T11:26:44.398 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.412+0000 7fc9e6ffd640 1 -- 172.21.15.86:0/241891431 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7fc9f00523f0 con 0x7fc9f413f650 2024-03-29T11:26:44.398 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.412+0000 7fc9e6ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:44.398 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.412+0000 7fc9e6ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:44.399 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:26:44.399 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:44.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.415+0000 7fc9fc6d6640 10 librados: watch_flush enter 2024-03-29T11:26:44.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.415+0000 7fc9fc6d6640 10 librados: watch_flush exit 2024-03-29T11:26:44.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.415+0000 7fc9fc6d6640 20 client.4484.objecter shutdown clearing up homeless session... 2024-03-29T11:26:44.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.415+0000 7fc9fc6d6640 10 client.4484.objecter successfully canceled tick 2024-03-29T11:26:44.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.416+0000 7fc9fc6d6640 1 -- 172.21.15.86:0/241891431 >> v1:172.21.15.196:6816/588110318 conn(0x7fc9d806dc40 legacy=0x7fc9d8070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:44.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.416+0000 7fc9fc6d6640 10 monclient: shutdown 2024-03-29T11:26:44.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.416+0000 7fc9fc6d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:44.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.416+0000 7fc9fc6d6640 1 -- 172.21.15.86:0/241891431 >> v1:172.21.15.86:6790/0 conn(0x7fc9f413f650 legacy=0x7fc9f4139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:44.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.416+0000 7fc9fc6d6640 1 -- 172.21.15.86:0/241891431 shutdown_connections 2024-03-29T11:26:44.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.416+0000 7fc9fc6d6640 1 -- 172.21.15.86:0/241891431 >> 172.21.15.86:0/241891431 conn(0x7fc9f4066a00 msgr2=0x7fc9f40b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:44.404 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.416+0000 7fc9fc6d6640 1 -- 172.21.15.86:0/241891431 shutdown_connections 2024-03-29T11:26:44.404 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.416+0000 7fc9fc6d6640 1 -- 172.21.15.86:0/241891431 wait complete. 2024-03-29T11:26:44.404 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:44.417+0000 7fc9fc6d6640 1 librados: shutdown 2024-03-29T11:26:44.423 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:51.231 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:51.388 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.401+0000 7f68b78c1640 10 monclient: get_monmap_and_config 2024-03-29T11:26:51.388 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.401+0000 7f68b78c1640 10 monclient: build_initial_monmap 2024-03-29T11:26:51.388 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.401+0000 7f68b78c1640 10 monclient: monmap: 2024-03-29T11:26:51.388 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:26:51.388 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:51.388 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:26:51.402323+0000 2024-03-29T11:26:51.389 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:26:51.402323+0000 2024-03-29T11:26:51.389 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:51.389 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:51.389 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:26:51.389 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:26:51.389 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:26:51.389 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:51.389 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.402+0000 7f68b78c1640 1 Processor -- start 2024-03-29T11:26:51.389 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.402+0000 7f68b78c1640 1 -- start start 2024-03-29T11:26:51.390 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.402+0000 7f68b78c1640 10 monclient: init 2024-03-29T11:26:51.390 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.403+0000 7f68b78c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:51.391 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.403+0000 7f68b78c1640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:26:51.391 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.403+0000 7f68b78c1640 10 monclient(hunting): picked mon.noname-b con 0x7f68b00b0860 addr v1:172.21.15.86:6790/0 2024-03-29T11:26:51.391 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.404+0000 7f68b78c1640 10 monclient(hunting): picked mon.noname-a con 0x7f68b0072450 addr v1:172.21.15.86:6789/0 2024-03-29T11:26:51.391 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.404+0000 7f68b78c1640 10 monclient(hunting): picked mon.noname-c con 0x7f68b013ee40 addr v1:172.21.15.196:6789/0 2024-03-29T11:26:51.391 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.404+0000 7f68b78c1640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b00ad840 con 0x7f68b0072450 2024-03-29T11:26:51.391 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.404+0000 7f68b78c1640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b013c830 con 0x7f68b013ee40 2024-03-29T11:26:51.391 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.404+0000 7f68b78c1640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b013c9c0 con 0x7f68b00b0860 2024-03-29T11:26:51.391 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.404+0000 7f68b78c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:51.391 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.404+0000 7f68b78c1640 10 monclient(hunting): authenticate will time out at 1235.722253s 2024-03-29T11:26:51.391 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.404+0000 7f68b565d640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f68b00b0860 0x7f68b00acfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:57940/0 (socket says 172.21.15.86:57940) 2024-03-29T11:26:51.391 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.404+0000 7f68b565d640 1 -- 172.21.15.86:0/782869673 learned_addr learned my addr 172.21.15.86:0/782869673 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:26:51.392 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.404+0000 7f68a7fff640 1 -- 172.21.15.86:0/782869673 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3206558546 0 0) 0x7f68b013c9c0 con 0x7f68b00b0860 2024-03-29T11:26:51.392 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.404+0000 7f68a7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:51.393 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.404+0000 7f68a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:51.393 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.404+0000 7f68a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:51.393 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.404+0000 7f68a7fff640 10 monclient(hunting): my global_id is 4487 2024-03-29T11:26:51.393 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.404+0000 7f68a7fff640 1 -- 172.21.15.86:0/782869673 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f689c005730 con 0x7f68b00b0860 2024-03-29T11:26:51.393 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.404+0000 7f68a7fff640 1 -- 172.21.15.86:0/782869673 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1888290552 0 0) 0x7f68b013c830 con 0x7f68b013ee40 2024-03-29T11:26:51.393 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.404+0000 7f68a7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:26:51.393 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.404+0000 7f68a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:51.393 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.404+0000 7f68a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:51.393 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.404+0000 7f68a7fff640 10 monclient(hunting): my global_id is 4420 2024-03-29T11:26:51.393 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.404+0000 7f68a7fff640 1 -- 172.21.15.86:0/782869673 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68b013c9c0 con 0x7f68b013ee40 2024-03-29T11:26:51.393 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.405+0000 7f68a7fff640 1 -- 172.21.15.86:0/782869673 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3278904074 0 0) 0x7f68b013c9c0 con 0x7f68b013ee40 2024-03-29T11:26:51.393 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.405+0000 7f68a7fff640 1 -- 172.21.15.86:0/782869673 >> v1:172.21.15.86:6790/0 conn(0x7f68b00b0860 legacy=0x7f68b00acfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:51.393 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.405+0000 7f68a7fff640 1 -- 172.21.15.86:0/782869673 >> v1:172.21.15.86:6789/0 conn(0x7f68b0072450 legacy=0x7f68b013c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:51.393 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.405+0000 7f68a7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:26:51.394 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.405+0000 7f68a7fff640 1 monclient: found mon.noname-c 2024-03-29T11:26:51.394 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.405+0000 7f68a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:51.394 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.405+0000 7f68a7fff640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T11:26:51.394 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.405+0000 7f68a7fff640 1 -- 172.21.15.86:0/782869673 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b013f970 con 0x7f68b013ee40 2024-03-29T11:26:51.394 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.405+0000 7f68a7fff640 10 monclient: _finish_auth 0 2024-03-29T11:26:51.394 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.405+0000 7f68a7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:26:51.394 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.405+0000 7f68a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:51.394 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.405+0000 7f68a7fff640 1 -- 172.21.15.86:0/782869673 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f68a8001920 con 0x7f68b013ee40 2024-03-29T11:26:51.395 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.405+0000 7f68a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:51.395 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.405+0000 7f68a7fff640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:51.395 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.405+0000 7f68a7fff640 10 monclient: dump: 2024-03-29T11:26:51.395 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:51.395 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:51.395 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:51.395 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:51.395 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:51.395 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:51.395 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:51.395 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:51.395 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:51.395 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:51.395 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.405+0000 7f68b78c1640 5 monclient: authenticate success, global_id 4420 2024-03-29T11:26:51.395 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.405+0000 7f68b78c1640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:51.395 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.405+0000 7f68a7fff640 1 -- 172.21.15.86:0/782869673 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68a8001b00 con 0x7f68b013ee40 2024-03-29T11:26:51.396 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.405+0000 7f68a7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:51.396 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.405+0000 7f68a7fff640 1 -- 172.21.15.86:0/782869673 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f68a8001ed0 con 0x7f68b013ee40 2024-03-29T11:26:51.397 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.405+0000 7f68a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:51.397 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.405+0000 7f68a7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:26:51.397 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.405+0000 7f68a7fff640 10 monclient: dump: 2024-03-29T11:26:51.398 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:51.398 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:51.398 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:51.398 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:51.398 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:51.398 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:51.398 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:51.398 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:51.398 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:51.398 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:51.398 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.406+0000 7f68b78c1640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:51.398 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.406+0000 7f68b78c1640 4 set_mon_vals no callback set 2024-03-29T11:26:51.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.406+0000 7f68b78c1640 10 monclient: shutdown 2024-03-29T11:26:51.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.406+0000 7f68b78c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:51.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.406+0000 7f68b78c1640 1 -- 172.21.15.86:0/782869673 >> v1:172.21.15.196:6789/0 conn(0x7f68b013ee40 legacy=0x7f68b013f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:51.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.407+0000 7f68b78c1640 1 -- 172.21.15.86:0/782869673 shutdown_connections 2024-03-29T11:26:51.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.407+0000 7f68b78c1640 1 -- 172.21.15.86:0/782869673 >> 172.21.15.86:0/782869673 conn(0x7f68b006b660 msgr2=0x7f68b006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:51.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.407+0000 7f68b78c1640 1 -- 172.21.15.86:0/782869673 shutdown_connections 2024-03-29T11:26:51.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.407+0000 7f68b78c1640 1 -- 172.21.15.86:0/782869673 wait complete. 2024-03-29T11:26:51.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.408+0000 7f68b78c1640 10 monclient: build_initial_monmap 2024-03-29T11:26:51.404 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.408+0000 7f68b78c1640 10 monclient: monmap: 2024-03-29T11:26:51.404 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:26:51.404 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:51.404 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:26:51.404 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:26:51.404 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:51.404 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:51.405 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:26:51.405 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:26:51.405 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:26:51.405 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:51.405 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.409+0000 7f68b78c1640 1 librados: starting msgr at 2024-03-29T11:26:51.405 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.409+0000 7f68b78c1640 1 librados: starting objecter 2024-03-29T11:26:51.405 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.409+0000 7f68b78c1640 1 Processor -- start 2024-03-29T11:26:51.405 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.409+0000 7f68b78c1640 1 -- start start 2024-03-29T11:26:51.405 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.409+0000 7f68b78c1640 1 librados: setting wanted keys 2024-03-29T11:26:51.405 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.409+0000 7f68b78c1640 1 librados: calling monclient init 2024-03-29T11:26:51.405 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.409+0000 7f68b78c1640 10 monclient: init 2024-03-29T11:26:51.405 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.410+0000 7f68b78c1640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:51.405 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.410+0000 7f68b78c1640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:26:51.405 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.410+0000 7f68b78c1640 10 monclient(hunting): picked mon.noname-a con 0x7f68b0072450 addr v1:172.21.15.86:6789/0 2024-03-29T11:26:51.406 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.410+0000 7f68b565d640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f68b0072450 0x7f68b01d7180 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:36408/0 (socket says 172.21.15.86:36408) 2024-03-29T11:26:51.406 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.410+0000 7f68b565d640 1 -- 172.21.15.86:0/4182937555 learned_addr learned my addr 172.21.15.86:0/4182937555 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:26:51.406 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.410+0000 7f68b78c1640 10 monclient(hunting): picked mon.noname-c con 0x7f68b00b0860 addr v1:172.21.15.86:6790/0 2024-03-29T11:26:51.406 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.410+0000 7f68a5ffb640 10 client.?.objecter ms_handle_connect 0x7f68b0072450 2024-03-29T11:26:51.406 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.410+0000 7f68a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:51.406 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.410+0000 7f68b78c1640 10 monclient(hunting): picked mon.noname-b con 0x7f68b013ee40 addr v1:172.21.15.196:6789/0 2024-03-29T11:26:51.406 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.411+0000 7f68b78c1640 1 -- 172.21.15.86:0/4182937555 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b01d6930 con 0x7f68b0072450 2024-03-29T11:26:51.406 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.411+0000 7f68b78c1640 1 -- 172.21.15.86:0/4182937555 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b006d9d0 con 0x7f68b013ee40 2024-03-29T11:26:51.406 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.411+0000 7f68b78c1640 1 -- 172.21.15.86:0/4182937555 --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b006db60 con 0x7f68b00b0860 2024-03-29T11:26:51.406 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.411+0000 7f68b78c1640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:51.406 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.411+0000 7f68b78c1640 10 monclient(hunting): authenticate will time out at 1235.729189s 2024-03-29T11:26:51.406 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.411+0000 7f68a5ffb640 10 client.?.objecter ms_handle_connect 0x7f68b00b0860 2024-03-29T11:26:51.406 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.411+0000 7f68a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:51.406 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.411+0000 7f68a5ffb640 10 client.?.objecter ms_handle_connect 0x7f68b013ee40 2024-03-29T11:26:51.407 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.411+0000 7f68a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:51.407 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.411+0000 7f68a5ffb640 1 -- 172.21.15.86:0/4182937555 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1182944082 0 0) 0x7f68b01d6930 con 0x7f68b0072450 2024-03-29T11:26:51.407 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.411+0000 7f68a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:51.407 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.411+0000 7f68a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:51.407 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.411+0000 7f68a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:51.407 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.411+0000 7f68a5ffb640 10 monclient(hunting): my global_id is 4434 2024-03-29T11:26:51.407 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.411+0000 7f68a5ffb640 1 -- 172.21.15.86:0/4182937555 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6894004eb0 con 0x7f68b0072450 2024-03-29T11:26:51.407 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.411+0000 7f68a5ffb640 1 -- 172.21.15.86:0/4182937555 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3261413476 0 0) 0x7f68b006db60 con 0x7f68b00b0860 2024-03-29T11:26:51.407 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.411+0000 7f68a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:51.407 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.411+0000 7f68a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:51.407 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.411+0000 7f68a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:51.407 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.411+0000 7f68a5ffb640 10 monclient(hunting): my global_id is 4490 2024-03-29T11:26:51.407 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.411+0000 7f68a5ffb640 1 -- 172.21.15.86:0/4182937555 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68b01d6930 con 0x7f68b00b0860 2024-03-29T11:26:51.407 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.411+0000 7f68a5ffb640 1 -- 172.21.15.86:0/4182937555 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2571588016 0 0) 0x7f68b006d9d0 con 0x7f68b013ee40 2024-03-29T11:26:51.407 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.411+0000 7f68a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:26:51.407 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.411+0000 7f68a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:51.408 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.411+0000 7f68a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:51.408 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.411+0000 7f68a5ffb640 10 monclient(hunting): my global_id is 4423 2024-03-29T11:26:51.408 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.411+0000 7f68a5ffb640 1 -- 172.21.15.86:0/4182937555 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f68b006db60 con 0x7f68b013ee40 2024-03-29T11:26:51.408 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.412+0000 7f68a5ffb640 1 -- 172.21.15.86:0/4182937555 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4213257988 0 0) 0x7f68b01d6930 con 0x7f68b00b0860 2024-03-29T11:26:51.408 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.412+0000 7f68a5ffb640 1 -- 172.21.15.86:0/4182937555 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68b006d9d0 con 0x7f68b00b0860 2024-03-29T11:26:51.408 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.412+0000 7f68a5ffb640 1 -- 172.21.15.86:0/4182937555 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f68a0001830 con 0x7f68b00b0860 2024-03-29T11:26:51.408 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.412+0000 7f68a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:51.408 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.412+0000 7f68a5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:51.408 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.412+0000 7f68a5ffb640 10 monclient(hunting): dump: 2024-03-29T11:26:51.408 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:51.408 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:51.408 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:51.408 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:51.408 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:51.408 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:51.409 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:51.409 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:51.409 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:51.409 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:51.409 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.412+0000 7f68a5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:51.409 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.412+0000 7f68a5ffb640 1 -- 172.21.15.86:0/4182937555 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 289179288 0 0) 0x7f6894004eb0 con 0x7f68b0072450 2024-03-29T11:26:51.409 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.412+0000 7f68a5ffb640 1 -- 172.21.15.86:0/4182937555 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f68b01d6930 con 0x7f68b0072450 2024-03-29T11:26:51.409 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.412+0000 7f68a5ffb640 1 -- 172.21.15.86:0/4182937555 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f68ac001950 con 0x7f68b0072450 2024-03-29T11:26:51.409 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.412+0000 7f68a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:51.409 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.412+0000 7f68a5ffb640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:26:51.409 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.412+0000 7f68a5ffb640 10 monclient(hunting): dump: 2024-03-29T11:26:51.409 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:51.409 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:51.409 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:51.410 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:51.410 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:51.410 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:51.410 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:51.410 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:51.410 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:51.410 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:51.410 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.412+0000 7f68a5ffb640 1 -- 172.21.15.86:0/4182937555 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 156666906 0 0) 0x7f68b006d9d0 con 0x7f68b00b0860 2024-03-29T11:26:51.410 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.412+0000 7f68a5ffb640 1 -- 172.21.15.86:0/4182937555 >> v1:172.21.15.196:6789/0 conn(0x7f68b013ee40 legacy=0x7f68b01dec90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:51.410 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.412+0000 7f68a5ffb640 1 -- 172.21.15.86:0/4182937555 >> v1:172.21.15.86:6789/0 conn(0x7f68b0072450 legacy=0x7f68b01d7180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:51.410 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.412+0000 7f68a5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:26:51.410 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.412+0000 7f68a5ffb640 1 monclient: found mon.c 2024-03-29T11:26:51.410 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.412+0000 7f68a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:51.410 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.412+0000 7f68a5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:26:51.410 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.412+0000 7f68a5ffb640 1 -- 172.21.15.86:0/4182937555 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b0071130 con 0x7f68b00b0860 2024-03-29T11:26:51.411 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.412+0000 7f68a5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:26:51.411 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.412+0000 7f68a5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:26:51.411 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.412+0000 7f68a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:51.411 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.412+0000 7f68b78c1640 5 monclient: authenticate success, global_id 4490 2024-03-29T11:26:51.411 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.412+0000 7f68b78c1640 10 monclient: _renew_subs 2024-03-29T11:26:51.411 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.412+0000 7f68b78c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:26:51.411 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.412+0000 7f68b78c1640 1 -- 172.21.15.86:0/4182937555 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68b00713e0 con 0x7f68b00b0860 2024-03-29T11:26:51.411 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.413+0000 7f68b78c1640 10 client.4490.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:51.411 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.413+0000 7f68b78c1640 10 monclient: _renew_subs 2024-03-29T11:26:51.411 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.413+0000 7f68b78c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:26:51.411 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.413+0000 7f68b78c1640 1 -- 172.21.15.86:0/4182937555 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f68b0142e10 con 0x7f68b00b0860 2024-03-29T11:26:51.411 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.413+0000 7f68b78c1640 1 librados: init done 2024-03-29T11:26:51.411 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.413+0000 7f68a5ffb640 1 -- 172.21.15.86:0/4182937555 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f68a0001d20 con 0x7f68b00b0860 2024-03-29T11:26:51.411 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.413+0000 7f68a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:51.411 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.413+0000 7f68a5ffb640 1 -- 172.21.15.86:0/4182937555 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f68a0003790 con 0x7f68b00b0860 2024-03-29T11:26:51.412 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.413+0000 7f68a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:51.412 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.413+0000 7f68a5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:51.412 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.413+0000 7f68a5ffb640 10 monclient: dump: 2024-03-29T11:26:51.412 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:51.412 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:51.412 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:51.412 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:51.412 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:51.412 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:51.412 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:51.412 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:51.412 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:51.412 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:51.412 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.413+0000 7f68a77fe640 4 set_mon_vals no callback set 2024-03-29T11:26:51.412 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.415+0000 7f68b78c1640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:51.412 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.415+0000 7f68b78c1640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:51.413 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.415+0000 7f68b78c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:26:51.413 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.415+0000 7f68b78c1640 1 -- 172.21.15.86:0/4182937555 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6890005080 con 0x7f68b00b0860 2024-03-29T11:26:51.413 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.416+0000 7f68a5ffb640 1 -- 172.21.15.86:0/4182937555 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f68a00038f0 con 0x7f68b00b0860 2024-03-29T11:26:51.413 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.417+0000 7f68a5ffb640 1 -- 172.21.15.86:0/4182937555 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f68a00836e0 con 0x7f68b00b0860 2024-03-29T11:26:51.413 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.417+0000 7f68a5ffb640 10 client.4490.objecter ms_dispatch 0x7f68b0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:51.414 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.417+0000 7f68a5ffb640 3 client.4490.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:51.414 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.417+0000 7f68a5ffb640 3 client.4490.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:51.414 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.417+0000 7f68a5ffb640 20 client.4490.objecter dump_active .. 0 homeless 2024-03-29T11:26:51.414 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.417+0000 7f68b565d640 10 monclient: get_auth_request con 0x7f6894072340 auth_method 0 2024-03-29T11:26:51.414 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.417+0000 7f68a5ffb640 10 client.4490.objecter ms_handle_connect 0x7f6894072340 2024-03-29T11:26:51.414 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.427+0000 7f68a5ffb640 1 -- 172.21.15.86:0/4182937555 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f68a004edf0 con 0x7f68b00b0860 2024-03-29T11:26:51.414 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.427+0000 7f68a5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:51.414 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.427+0000 7f68a5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:51.743 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.756+0000 7f68b78c1640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:51.743 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.756+0000 7f68b78c1640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:51.743 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.756+0000 7f68b78c1640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:26:51.743 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.756+0000 7f68b78c1640 1 -- 172.21.15.86:0/4182937555 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6890005330 con 0x7f68b00b0860 2024-03-29T11:26:51.744 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.757+0000 7f68a5ffb640 1 -- 172.21.15.86:0/4182937555 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f68a0050ef0 con 0x7f68b00b0860 2024-03-29T11:26:51.744 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.757+0000 7f68a5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:51.744 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.757+0000 7f68a5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:51.744 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:26:51.744 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:51.747 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.760+0000 7f68b78c1640 10 librados: watch_flush enter 2024-03-29T11:26:51.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.760+0000 7f68b78c1640 10 librados: watch_flush exit 2024-03-29T11:26:51.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.760+0000 7f68b78c1640 20 client.4490.objecter shutdown clearing up homeless session... 2024-03-29T11:26:51.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.760+0000 7f68b78c1640 10 client.4490.objecter successfully canceled tick 2024-03-29T11:26:51.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.760+0000 7f68b78c1640 1 -- 172.21.15.86:0/4182937555 >> v1:172.21.15.196:6816/588110318 conn(0x7f6894072340 legacy=0x7f6894074800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:51.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.760+0000 7f68b78c1640 10 monclient: shutdown 2024-03-29T11:26:51.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.760+0000 7f68b78c1640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:51.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.760+0000 7f68b78c1640 1 -- 172.21.15.86:0/4182937555 >> v1:172.21.15.86:6790/0 conn(0x7f68b00b0860 legacy=0x7f68b01356b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:51.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.760+0000 7f68b78c1640 1 -- 172.21.15.86:0/4182937555 shutdown_connections 2024-03-29T11:26:51.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.760+0000 7f68b78c1640 1 -- 172.21.15.86:0/4182937555 >> 172.21.15.86:0/4182937555 conn(0x7f68b006b660 msgr2=0x7f68b006ba40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:51.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.761+0000 7f68b78c1640 1 -- 172.21.15.86:0/4182937555 shutdown_connections 2024-03-29T11:26:51.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.761+0000 7f68b78c1640 1 -- 172.21.15.86:0/4182937555 wait complete. 2024-03-29T11:26:51.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:51.761+0000 7f68b78c1640 1 librados: shutdown 2024-03-29T11:26:51.762 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:26:58.571 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:26:58.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.758+0000 7f3028915640 10 monclient: get_monmap_and_config 2024-03-29T11:26:58.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.758+0000 7f3028915640 10 monclient: build_initial_monmap 2024-03-29T11:26:58.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.758+0000 7f3028915640 10 monclient: monmap: 2024-03-29T11:26:58.748 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:26:58.748 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:58.748 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:26:58.759431+0000 2024-03-29T11:26:58.748 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:26:58.759431+0000 2024-03-29T11:26:58.748 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:58.748 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:58.748 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:26:58.748 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:26:58.748 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:26:58.749 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:58.749 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.759+0000 7f3028915640 1 Processor -- start 2024-03-29T11:26:58.749 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.759+0000 7f3028915640 1 -- start start 2024-03-29T11:26:58.749 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.759+0000 7f3028915640 10 monclient: init 2024-03-29T11:26:58.749 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.760+0000 7f3028915640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:58.749 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.760+0000 7f3028915640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:26:58.749 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.760+0000 7f3028915640 10 monclient(hunting): picked mon.noname-a con 0x7f302406ea10 addr v1:172.21.15.86:6789/0 2024-03-29T11:26:58.749 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.760+0000 7f3028915640 10 monclient(hunting): picked mon.noname-b con 0x7f302413bd30 addr v1:172.21.15.86:6790/0 2024-03-29T11:26:58.749 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.760+0000 7f3028915640 10 monclient(hunting): picked mon.noname-c con 0x7f302406edf0 addr v1:172.21.15.196:6789/0 2024-03-29T11:26:58.749 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.760+0000 7f3028915640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3024070f40 con 0x7f302406ea10 2024-03-29T11:26:58.749 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.760+0000 7f3028915640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3024071360 con 0x7f302406edf0 2024-03-29T11:26:58.749 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.760+0000 7f3028915640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3024072e60 con 0x7f302413bd30 2024-03-29T11:26:58.749 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.760+0000 7f3028915640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:58.749 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.760+0000 7f3028915640 10 monclient(hunting): authenticate will time out at 1243.078662s 2024-03-29T11:26:58.753 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.766+0000 7f3022d9d640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f302406edf0 0x7f3024149d10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:45437/0 (socket says 172.21.15.86:45437) 2024-03-29T11:26:58.754 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.766+0000 7f3022d9d640 1 -- 172.21.15.86:0/3221104915 learned_addr learned my addr 172.21.15.86:0/3221104915 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:26:58.756 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.769+0000 7f3021d9b640 1 -- 172.21.15.86:0/3221104915 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2460418157 0 0) 0x7f3024071360 con 0x7f302406edf0 2024-03-29T11:26:58.756 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.769+0000 7f3021d9b640 10 monclient(hunting): _negotiate 2024-03-29T11:26:58.756 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.769+0000 7f3021d9b640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:58.756 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.769+0000 7f3021d9b640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:58.756 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.769+0000 7f3021d9b640 10 monclient(hunting): my global_id is 4426 2024-03-29T11:26:58.756 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.769+0000 7f3021d9b640 1 -- 172.21.15.86:0/3221104915 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3010003440 con 0x7f302406edf0 2024-03-29T11:26:58.759 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.769+0000 7f3021d9b640 1 -- 172.21.15.86:0/3221104915 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 929439756 0 0) 0x7f3010003440 con 0x7f302406edf0 2024-03-29T11:26:58.759 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.769+0000 7f3021d9b640 1 -- 172.21.15.86:0/3221104915 >> v1:172.21.15.86:6790/0 conn(0x7f302413bd30 legacy=0x7f302413e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:58.759 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.769+0000 7f3021d9b640 1 -- 172.21.15.86:0/3221104915 >> v1:172.21.15.86:6789/0 conn(0x7f302406ea10 legacy=0x7f302413b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:26:58.759 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.769+0000 7f3021d9b640 10 monclient: _finish_hunting 0 2024-03-29T11:26:58.759 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.769+0000 7f3021d9b640 1 monclient: found mon.noname-c 2024-03-29T11:26:58.759 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.769+0000 7f3021d9b640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:58.759 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.769+0000 7f3021d9b640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T11:26:58.759 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.769+0000 7f3021d9b640 1 -- 172.21.15.86:0/3221104915 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f302406fb00 con 0x7f302406edf0 2024-03-29T11:26:58.759 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.769+0000 7f3021d9b640 10 monclient: _finish_auth 0 2024-03-29T11:26:58.759 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.769+0000 7f3021d9b640 10 monclient: _check_auth_tickets 2024-03-29T11:26:58.759 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.769+0000 7f3021d9b640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:58.759 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.769+0000 7f3028915640 5 monclient: authenticate success, global_id 4426 2024-03-29T11:26:58.759 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.769+0000 7f3028915640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:58.759 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.769+0000 7f3021d9b640 1 -- 172.21.15.86:0/3221104915 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f301c001a10 con 0x7f302406edf0 2024-03-29T11:26:58.760 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.769+0000 7f3021d9b640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:58.760 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.769+0000 7f3021d9b640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:58.760 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.769+0000 7f3021d9b640 10 monclient: dump: 2024-03-29T11:26:58.760 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:58.760 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:58.760 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:58.760 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:58.760 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:58.760 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:58.760 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:58.760 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:58.760 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:58.760 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:58.760 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.769+0000 7f3028915640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:26:58.761 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.770+0000 7f3021d9b640 1 -- 172.21.15.86:0/3221104915 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f301c001be0 con 0x7f302406edf0 2024-03-29T11:26:58.761 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.770+0000 7f3021d9b640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:58.761 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.770+0000 7f3028915640 10 monclient: get_monmap_and_config success 2024-03-29T11:26:58.761 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.770+0000 7f3028915640 4 set_mon_vals no callback set 2024-03-29T11:26:58.761 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.771+0000 7f3028915640 10 monclient: shutdown 2024-03-29T11:26:58.761 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.771+0000 7f3028915640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:58.761 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.771+0000 7f3028915640 1 -- 172.21.15.86:0/3221104915 >> v1:172.21.15.196:6789/0 conn(0x7f302406edf0 legacy=0x7f3024149d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:58.761 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.771+0000 7f3028915640 1 -- 172.21.15.86:0/3221104915 shutdown_connections 2024-03-29T11:26:58.761 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.771+0000 7f3028915640 1 -- 172.21.15.86:0/3221104915 >> 172.21.15.86:0/3221104915 conn(0x7f30240b3f40 msgr2=0x7f30240b4320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:58.762 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.774+0000 7f3028915640 1 -- 172.21.15.86:0/3221104915 shutdown_connections 2024-03-29T11:26:58.762 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.774+0000 7f3028915640 1 -- 172.21.15.86:0/3221104915 wait complete. 2024-03-29T11:26:58.763 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.776+0000 7f3028915640 10 monclient: build_initial_monmap 2024-03-29T11:26:58.763 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.776+0000 7f3028915640 10 monclient: monmap: 2024-03-29T11:26:58.763 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:26:58.763 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:26:58.763 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:26:58.763 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:26:58.763 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:26:58.763 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:58.763 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:26:58.763 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:26:58.763 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:26:58.763 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:58.764 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.776+0000 7f3028915640 1 librados: starting msgr at 2024-03-29T11:26:58.764 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.776+0000 7f3028915640 1 librados: starting objecter 2024-03-29T11:26:58.764 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.776+0000 7f3028915640 1 Processor -- start 2024-03-29T11:26:58.764 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.776+0000 7f3028915640 1 -- start start 2024-03-29T11:26:58.764 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.776+0000 7f3028915640 1 librados: setting wanted keys 2024-03-29T11:26:58.765 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.776+0000 7f3028915640 1 librados: calling monclient init 2024-03-29T11:26:58.765 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.776+0000 7f3028915640 10 monclient: init 2024-03-29T11:26:58.765 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.777+0000 7f3028915640 10 monclient: _reopen_session rank -1 2024-03-29T11:26:58.765 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.777+0000 7f3028915640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:26:58.765 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.777+0000 7f3028915640 10 monclient(hunting): picked mon.noname-a con 0x7f302406ea10 addr v1:172.21.15.86:6789/0 2024-03-29T11:26:58.765 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.777+0000 7f3028915640 10 monclient(hunting): picked mon.noname-b con 0x7f302413fe40 addr v1:172.21.15.196:6789/0 2024-03-29T11:26:58.773 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.780+0000 7f3028915640 10 monclient(hunting): picked mon.noname-c con 0x7f30241408e0 addr v1:172.21.15.86:6790/0 2024-03-29T11:26:58.773 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.780+0000 7f3028915640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f302413c1f0 con 0x7f302406ea10 2024-03-29T11:26:58.773 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.780+0000 7f3028915640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30241d2710 con 0x7f302413fe40 2024-03-29T11:26:58.773 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.780+0000 7f3028915640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3024141a00 con 0x7f30241408e0 2024-03-29T11:26:58.773 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.780+0000 7f3028915640 10 monclient(hunting): _renew_subs 2024-03-29T11:26:58.773 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.780+0000 7f3028915640 10 monclient(hunting): authenticate will time out at 1243.098969s 2024-03-29T11:26:58.773 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.781+0000 7f3022d9d640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f30241408e0 0x7f30241e6df0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:36499/0 (socket says 172.21.15.86:36499) 2024-03-29T11:26:58.773 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.781+0000 7f3022d9d640 1 -- 172.21.15.86:0/1399381493 learned_addr learned my addr 172.21.15.86:0/1399381493 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:26:58.774 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.783+0000 7f301affd640 10 client.?.objecter ms_handle_connect 0x7f30241408e0 2024-03-29T11:26:58.774 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.784+0000 7f301affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:58.774 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.784+0000 7f301affd640 10 client.?.objecter ms_handle_connect 0x7f302413fe40 2024-03-29T11:26:58.774 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.784+0000 7f301affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:58.774 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.784+0000 7f301affd640 1 -- 172.21.15.86:0/1399381493 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1691449000 0 0) 0x7f3024141a00 con 0x7f30241408e0 2024-03-29T11:26:58.774 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.784+0000 7f301affd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:58.774 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.784+0000 7f301affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:58.774 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.784+0000 7f301affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:58.774 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.784+0000 7f301affd640 10 monclient(hunting): my global_id is 4493 2024-03-29T11:26:58.774 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.784+0000 7f301affd640 1 -- 172.21.15.86:0/1399381493 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3004004eb0 con 0x7f30241408e0 2024-03-29T11:26:58.774 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.784+0000 7f301affd640 10 client.?.objecter ms_handle_connect 0x7f302406ea10 2024-03-29T11:26:58.774 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.784+0000 7f301affd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:26:58.774 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.785+0000 7f301affd640 1 -- 172.21.15.86:0/1399381493 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4175174916 0 0) 0x7f3004004eb0 con 0x7f30241408e0 2024-03-29T11:26:58.774 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.785+0000 7f301affd640 1 -- 172.21.15.86:0/1399381493 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3024141a00 con 0x7f30241408e0 2024-03-29T11:26:58.775 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.785+0000 7f301affd640 1 -- 172.21.15.86:0/1399381493 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f301c004b60 con 0x7f30241408e0 2024-03-29T11:26:58.775 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.785+0000 7f301affd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:58.775 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.785+0000 7f301affd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:26:58.775 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.785+0000 7f301affd640 10 monclient(hunting): dump: 2024-03-29T11:26:58.775 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:58.775 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:58.775 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:58.775 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:58.775 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:58.775 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:58.775 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:58.775 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:58.775 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:58.775 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:58.775 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.785+0000 7f301affd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:26:58.776 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.785+0000 7f301affd640 1 -- 172.21.15.86:0/1399381493 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 77573232 0 0) 0x7f30241d2710 con 0x7f302413fe40 2024-03-29T11:26:58.776 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.785+0000 7f301affd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:58.776 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.785+0000 7f301affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:58.776 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.785+0000 7f301affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:58.776 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.785+0000 7f301affd640 10 monclient(hunting): my global_id is 4429 2024-03-29T11:26:58.776 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.785+0000 7f301affd640 1 -- 172.21.15.86:0/1399381493 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3004004eb0 con 0x7f302413fe40 2024-03-29T11:26:58.779 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.787+0000 7f301affd640 1 -- 172.21.15.86:0/1399381493 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3321085128 0 0) 0x7f302413c1f0 con 0x7f302406ea10 2024-03-29T11:26:58.779 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.787+0000 7f301affd640 10 monclient(hunting): _negotiate 2024-03-29T11:26:58.779 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.787+0000 7f301affd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:26:58.779 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.787+0000 7f301affd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:26:58.780 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.787+0000 7f301affd640 10 monclient(hunting): my global_id is 4437 2024-03-29T11:26:58.780 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.787+0000 7f301affd640 1 -- 172.21.15.86:0/1399381493 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f30241d2710 con 0x7f302406ea10 2024-03-29T11:26:58.782 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.787+0000 7f301affd640 1 -- 172.21.15.86:0/1399381493 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 295113464 0 0) 0x7f3024141a00 con 0x7f30241408e0 2024-03-29T11:26:58.782 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.787+0000 7f301affd640 1 -- 172.21.15.86:0/1399381493 >> v1:172.21.15.196:6789/0 conn(0x7f302413fe40 legacy=0x7f302413c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:58.782 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.787+0000 7f301affd640 1 -- 172.21.15.86:0/1399381493 >> v1:172.21.15.86:6789/0 conn(0x7f302406ea10 legacy=0x7f302413f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:58.782 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.787+0000 7f301affd640 10 monclient: _finish_hunting 0 2024-03-29T11:26:58.782 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.787+0000 7f301affd640 1 monclient: found mon.c 2024-03-29T11:26:58.782 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.787+0000 7f301affd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:26:58.782 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.787+0000 7f301affd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:26:58.782 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.787+0000 7f301affd640 1 -- 172.21.15.86:0/1399381493 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30241d2410 con 0x7f30241408e0 2024-03-29T11:26:58.782 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.787+0000 7f301affd640 10 monclient: _finish_auth 0 2024-03-29T11:26:58.782 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.787+0000 7f301affd640 10 monclient: _check_auth_tickets 2024-03-29T11:26:58.782 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.787+0000 7f301affd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:26:58.782 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.788+0000 7f3028915640 5 monclient: authenticate success, global_id 4493 2024-03-29T11:26:58.782 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.788+0000 7f3028915640 10 monclient: _renew_subs 2024-03-29T11:26:58.782 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.788+0000 7f3028915640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:26:58.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.788+0000 7f3028915640 1 -- 172.21.15.86:0/1399381493 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30240ae260 con 0x7f30241408e0 2024-03-29T11:26:58.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.788+0000 7f3028915640 10 client.4493.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:26:58.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.788+0000 7f3028915640 10 monclient: _renew_subs 2024-03-29T11:26:58.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.788+0000 7f3028915640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:26:58.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.788+0000 7f3028915640 1 -- 172.21.15.86:0/1399381493 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f30241425e0 con 0x7f30241408e0 2024-03-29T11:26:58.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.788+0000 7f3028915640 1 librados: init done 2024-03-29T11:26:58.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.789+0000 7f301affd640 1 -- 172.21.15.86:0/1399381493 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f301c0015f0 con 0x7f30241408e0 2024-03-29T11:26:58.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.789+0000 7f301affd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:26:58.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.789+0000 7f301affd640 1 -- 172.21.15.86:0/1399381493 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f301c005030 con 0x7f30241408e0 2024-03-29T11:26:58.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.789+0000 7f301affd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:26:58.783 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.789+0000 7f301affd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:26:58.784 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.789+0000 7f301affd640 10 monclient: dump: 2024-03-29T11:26:58.784 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:26:58.784 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:26:58.784 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:58.784 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:26:58.784 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:26:58.784 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:26:58.785 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:26:58.785 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:26:58.785 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:26:58.785 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:26:58.785 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.789+0000 7f3020d99640 4 set_mon_vals no callback set 2024-03-29T11:26:58.785 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.790+0000 7f3028915640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:58.785 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.791+0000 7f3028915640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:58.785 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.791+0000 7f3028915640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:26:58.785 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.791+0000 7f3028915640 1 -- 172.21.15.86:0/1399381493 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3008005080 con 0x7f30241408e0 2024-03-29T11:26:58.786 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.795+0000 7f301affd640 1 -- 172.21.15.86:0/1399381493 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f301c005190 con 0x7f30241408e0 2024-03-29T11:26:58.786 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.796+0000 7f302259c640 10 monclient: get_auth_request con 0x7f300406def0 auth_method 0 2024-03-29T11:26:58.786 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.796+0000 7f301affd640 1 -- 172.21.15.86:0/1399381493 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f301c084e80 con 0x7f30241408e0 2024-03-29T11:26:58.787 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.796+0000 7f301affd640 10 client.4493.objecter ms_dispatch 0x7f3024000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:26:58.787 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.796+0000 7f301affd640 3 client.4493.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:26:58.787 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.796+0000 7f301affd640 3 client.4493.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:26:58.787 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.796+0000 7f301affd640 20 client.4493.objecter dump_active .. 0 homeless 2024-03-29T11:26:58.787 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.796+0000 7f301affd640 10 client.4493.objecter ms_handle_connect 0x7f300406def0 2024-03-29T11:26:58.788 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.801+0000 7f301affd640 1 -- 172.21.15.86:0/1399381493 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f301c050760 con 0x7f30241408e0 2024-03-29T11:26:58.788 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.801+0000 7f301affd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:26:58.788 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:58.801+0000 7f301affd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:26:59.140 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:59.153+0000 7f3028915640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:59.141 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:59.153+0000 7f3028915640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:59.141 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:59.153+0000 7f3028915640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:26:59.141 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:59.153+0000 7f3028915640 1 -- 172.21.15.86:0/1399381493 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3008005330 con 0x7f30241408e0 2024-03-29T11:26:59.141 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:59.154+0000 7f301affd640 1 -- 172.21.15.86:0/1399381493 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f301c052690 con 0x7f30241408e0 2024-03-29T11:26:59.141 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:59.154+0000 7f301affd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:26:59.141 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:59.154+0000 7f301affd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:26:59.141 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:26:59.142 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:26:59.146 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:59.158+0000 7f3028915640 10 librados: watch_flush enter 2024-03-29T11:26:59.146 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:59.158+0000 7f3028915640 10 librados: watch_flush exit 2024-03-29T11:26:59.146 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:59.158+0000 7f3028915640 20 client.4493.objecter shutdown clearing up homeless session... 2024-03-29T11:26:59.147 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:59.158+0000 7f3028915640 10 client.4493.objecter successfully canceled tick 2024-03-29T11:26:59.147 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:59.158+0000 7f3028915640 1 -- 172.21.15.86:0/1399381493 >> v1:172.21.15.196:6816/588110318 conn(0x7f300406def0 legacy=0x7f30040703b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:59.147 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:59.158+0000 7f3028915640 10 monclient: shutdown 2024-03-29T11:26:59.147 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:59.158+0000 7f3028915640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:26:59.147 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:59.158+0000 7f3028915640 1 -- 172.21.15.86:0/1399381493 >> v1:172.21.15.86:6790/0 conn(0x7f30241408e0 legacy=0x7f30241e6df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:26:59.147 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:59.159+0000 7f3028915640 1 -- 172.21.15.86:0/1399381493 shutdown_connections 2024-03-29T11:26:59.147 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:59.159+0000 7f3028915640 1 -- 172.21.15.86:0/1399381493 >> 172.21.15.86:0/1399381493 conn(0x7f30240b3f40 msgr2=0x7f302413b3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:26:59.147 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:59.159+0000 7f3028915640 1 -- 172.21.15.86:0/1399381493 shutdown_connections 2024-03-29T11:26:59.147 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:59.159+0000 7f3028915640 1 -- 172.21.15.86:0/1399381493 wait complete. 2024-03-29T11:26:59.147 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:26:59.159+0000 7f3028915640 1 librados: shutdown 2024-03-29T11:26:59.161 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:05.969 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:06.119 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.131+0000 7fd432f2b640 10 monclient: get_monmap_and_config 2024-03-29T11:27:06.119 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.131+0000 7fd432f2b640 10 monclient: build_initial_monmap 2024-03-29T11:27:06.119 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.131+0000 7fd432f2b640 10 monclient: monmap: 2024-03-29T11:27:06.119 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:27:06.119 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:06.119 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:27:06.132739+0000 2024-03-29T11:27:06.119 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:27:06.132739+0000 2024-03-29T11:27:06.119 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:06.119 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:06.119 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:27:06.119 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:27:06.119 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:27:06.120 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:06.122 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.134+0000 7fd432f2b640 1 Processor -- start 2024-03-29T11:27:06.122 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.135+0000 7fd432f2b640 1 -- start start 2024-03-29T11:27:06.122 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.135+0000 7fd432f2b640 10 monclient: init 2024-03-29T11:27:06.123 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.135+0000 7fd432f2b640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:06.123 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.135+0000 7fd432f2b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:06.123 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.135+0000 7fd432f2b640 10 monclient(hunting): picked mon.noname-a con 0x7fd42c1457f0 addr v1:172.21.15.86:6789/0 2024-03-29T11:27:06.123 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.135+0000 7fd432f2b640 10 monclient(hunting): picked mon.noname-b con 0x7fd42c13f650 addr v1:172.21.15.86:6790/0 2024-03-29T11:27:06.123 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.135+0000 7fd432f2b640 10 monclient(hunting): picked mon.noname-c con 0x7fd42c0a9d70 addr v1:172.21.15.196:6789/0 2024-03-29T11:27:06.123 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.135+0000 7fd432f2b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd42c0aaef0 con 0x7fd42c1457f0 2024-03-29T11:27:06.123 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.135+0000 7fd432f2b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd42c145bd0 con 0x7fd42c0a9d70 2024-03-29T11:27:06.124 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.135+0000 7fd432f2b640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd42c145eb0 con 0x7fd42c13f650 2024-03-29T11:27:06.124 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.135+0000 7fd432f2b640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:06.124 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.135+0000 7fd432f2b640 10 monclient(hunting): authenticate will time out at 1250.453994s 2024-03-29T11:27:06.124 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.136+0000 7fd42bfff640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7fd42c13f650 0x7fd42c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:34444/0 (socket says 172.21.15.86:34444) 2024-03-29T11:27:06.124 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.136+0000 7fd42bfff640 1 -- 172.21.15.86:0/1221747 learned_addr learned my addr 172.21.15.86:0/1221747 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:27:06.124 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.136+0000 7fd42b7fe640 1 -- 172.21.15.86:0/1221747 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3883387005 0 0) 0x7fd42c145eb0 con 0x7fd42c13f650 2024-03-29T11:27:06.125 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.136+0000 7fd42b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:06.125 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.136+0000 7fd42b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:06.125 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.136+0000 7fd42b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:06.125 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.136+0000 7fd42b7fe640 10 monclient(hunting): my global_id is 4496 2024-03-29T11:27:06.125 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.136+0000 7fd42b7fe640 1 -- 172.21.15.86:0/1221747 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd414003520 con 0x7fd42c13f650 2024-03-29T11:27:06.125 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.137+0000 7fd42b7fe640 1 -- 172.21.15.86:0/1221747 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2891852752 0 0) 0x7fd414003520 con 0x7fd42c13f650 2024-03-29T11:27:06.125 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.137+0000 7fd42b7fe640 1 -- 172.21.15.86:0/1221747 >> v1:172.21.15.196:6789/0 conn(0x7fd42c0a9d70 legacy=0x7fd42c14d8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:06.125 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.137+0000 7fd42b7fe640 1 -- 172.21.15.86:0/1221747 >> v1:172.21.15.86:6789/0 conn(0x7fd42c1457f0 legacy=0x7fd42c13ef40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:06.126 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.138+0000 7fd42b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:27:06.126 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.138+0000 7fd42b7fe640 1 monclient: found mon.noname-b 2024-03-29T11:27:06.126 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.138+0000 7fd42b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:06.126 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.138+0000 7fd42b7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:27:06.126 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.138+0000 7fd42b7fe640 1 -- 172.21.15.86:0/1221747 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd42c146340 con 0x7fd42c13f650 2024-03-29T11:27:06.126 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.138+0000 7fd42b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:27:06.126 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.138+0000 7fd42b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:27:06.126 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.138+0000 7fd42b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:06.126 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.138+0000 7fd432f2b640 5 monclient: authenticate success, global_id 4496 2024-03-29T11:27:06.126 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.138+0000 7fd432f2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:06.126 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.138+0000 7fd42b7fe640 1 -- 172.21.15.86:0/1221747 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fd41c001690 con 0x7fd42c13f650 2024-03-29T11:27:06.127 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.138+0000 7fd42b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:06.127 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.138+0000 7fd42b7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:06.127 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.138+0000 7fd42b7fe640 10 monclient: dump: 2024-03-29T11:27:06.127 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:06.127 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:06.127 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:06.127 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:06.127 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:06.127 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:06.127 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:06.127 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:06.127 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:06.128 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:06.128 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.138+0000 7fd432f2b640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:06.128 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.138+0000 7fd42b7fe640 1 -- 172.21.15.86:0/1221747 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd41c001920 con 0x7fd42c13f650 2024-03-29T11:27:06.128 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.138+0000 7fd42b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:06.128 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.138+0000 7fd432f2b640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:06.128 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.138+0000 7fd432f2b640 4 set_mon_vals no callback set 2024-03-29T11:27:06.130 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.139+0000 7fd432f2b640 10 monclient: shutdown 2024-03-29T11:27:06.130 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.139+0000 7fd42b7fe640 1 -- 172.21.15.86:0/1221747 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fd41c001cb0 con 0x7fd42c13f650 2024-03-29T11:27:06.130 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.139+0000 7fd42b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:06.130 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.139+0000 7fd42b7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:06.130 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.139+0000 7fd42b7fe640 10 monclient: dump: 2024-03-29T11:27:06.131 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:06.131 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:06.131 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:06.131 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:06.131 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:06.131 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:06.131 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:06.131 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:06.131 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:06.131 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:06.131 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.139+0000 7fd432f2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:06.131 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.139+0000 7fd432f2b640 1 -- 172.21.15.86:0/1221747 >> v1:172.21.15.86:6790/0 conn(0x7fd42c13f650 legacy=0x7fd42c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:06.131 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.139+0000 7fd432f2b640 1 -- 172.21.15.86:0/1221747 shutdown_connections 2024-03-29T11:27:06.131 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.139+0000 7fd432f2b640 1 -- 172.21.15.86:0/1221747 >> 172.21.15.86:0/1221747 conn(0x7fd42c066a00 msgr2=0x7fd42c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:06.132 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.140+0000 7fd432f2b640 1 -- 172.21.15.86:0/1221747 shutdown_connections 2024-03-29T11:27:06.132 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.140+0000 7fd432f2b640 1 -- 172.21.15.86:0/1221747 wait complete. 2024-03-29T11:27:06.132 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.141+0000 7fd432f2b640 10 monclient: build_initial_monmap 2024-03-29T11:27:06.132 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.141+0000 7fd432f2b640 10 monclient: monmap: 2024-03-29T11:27:06.132 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:27:06.132 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:06.132 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:27:06.132 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:27:06.132 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:06.132 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:06.132 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:27:06.132 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:27:06.132 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:27:06.132 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:06.132 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.141+0000 7fd432f2b640 1 librados: starting msgr at 2024-03-29T11:27:06.132 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.141+0000 7fd432f2b640 1 librados: starting objecter 2024-03-29T11:27:06.133 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.141+0000 7fd432f2b640 1 Processor -- start 2024-03-29T11:27:06.133 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.141+0000 7fd432f2b640 1 -- start start 2024-03-29T11:27:06.133 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.141+0000 7fd432f2b640 1 librados: setting wanted keys 2024-03-29T11:27:06.133 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.141+0000 7fd432f2b640 1 librados: calling monclient init 2024-03-29T11:27:06.133 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.141+0000 7fd432f2b640 10 monclient: init 2024-03-29T11:27:06.133 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.142+0000 7fd432f2b640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:06.133 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.142+0000 7fd432f2b640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:06.133 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.142+0000 7fd432f2b640 10 monclient(hunting): picked mon.noname-a con 0x7fd42c0a9d70 addr v1:172.21.15.86:6789/0 2024-03-29T11:27:06.134 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.145+0000 7fd432f2b640 10 monclient(hunting): picked mon.noname-c con 0x7fd42c1457f0 addr v1:172.21.15.86:6790/0 2024-03-29T11:27:06.134 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.145+0000 7fd432f2b640 10 monclient(hunting): picked mon.noname-b con 0x7fd42c1447b0 addr v1:172.21.15.196:6789/0 2024-03-29T11:27:06.134 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.145+0000 7fd432f2b640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd42c06e6f0 con 0x7fd42c0a9d70 2024-03-29T11:27:06.134 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.145+0000 7fd432f2b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd42c06e960 con 0x7fd42c1447b0 2024-03-29T11:27:06.134 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.145+0000 7fd432f2b640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd42c06ec40 con 0x7fd42c1457f0 2024-03-29T11:27:06.134 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.145+0000 7fd432f2b640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:06.134 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.145+0000 7fd432f2b640 10 monclient(hunting): authenticate will time out at 1250.463732s 2024-03-29T11:27:06.134 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.145+0000 7fd42bfff640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7fd42c1457f0 0x7fd42c141ce0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:35200/0 (socket says 172.21.15.86:35200) 2024-03-29T11:27:06.134 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.145+0000 7fd42bfff640 1 -- 172.21.15.86:0/52323815 learned_addr learned my addr 172.21.15.86:0/52323815 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:27:06.134 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.145+0000 7fd4297fa640 10 client.?.objecter ms_handle_connect 0x7fd42c1457f0 2024-03-29T11:27:06.135 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.145+0000 7fd4297fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:06.136 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.147+0000 7fd4297fa640 10 client.?.objecter ms_handle_connect 0x7fd42c0a9d70 2024-03-29T11:27:06.136 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.147+0000 7fd4297fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:06.136 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.147+0000 7fd4297fa640 1 -- 172.21.15.86:0/52323815 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3390968404 0 0) 0x7fd42c06ec40 con 0x7fd42c1457f0 2024-03-29T11:27:06.136 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.147+0000 7fd4297fa640 10 monclient(hunting): _negotiate 2024-03-29T11:27:06.137 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.147+0000 7fd4297fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:06.137 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.147+0000 7fd4297fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:06.137 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.149+0000 7fd4297fa640 10 monclient(hunting): my global_id is 4499 2024-03-29T11:27:06.137 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.149+0000 7fd4297fa640 1 -- 172.21.15.86:0/52323815 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd418004eb0 con 0x7fd42c1457f0 2024-03-29T11:27:06.139 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.150+0000 7fd4297fa640 1 -- 172.21.15.86:0/52323815 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2079812305 0 0) 0x7fd42c06e6f0 con 0x7fd42c0a9d70 2024-03-29T11:27:06.139 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.150+0000 7fd4297fa640 10 monclient(hunting): _negotiate 2024-03-29T11:27:06.139 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.150+0000 7fd4297fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:06.139 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.150+0000 7fd4297fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:06.140 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.150+0000 7fd4297fa640 10 monclient(hunting): my global_id is 4440 2024-03-29T11:27:06.140 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.150+0000 7fd4297fa640 1 -- 172.21.15.86:0/52323815 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd42c06ec40 con 0x7fd42c0a9d70 2024-03-29T11:27:06.140 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.150+0000 7fd4297fa640 1 -- 172.21.15.86:0/52323815 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1925248720 0 0) 0x7fd418004eb0 con 0x7fd42c1457f0 2024-03-29T11:27:06.140 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.150+0000 7fd4297fa640 1 -- 172.21.15.86:0/52323815 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd42c06e6f0 con 0x7fd42c1457f0 2024-03-29T11:27:06.140 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.150+0000 7fd4297fa640 1 -- 172.21.15.86:0/52323815 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fd41c001610 con 0x7fd42c1457f0 2024-03-29T11:27:06.140 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.150+0000 7fd4297fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:06.140 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.150+0000 7fd4297fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:06.140 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.150+0000 7fd4297fa640 10 monclient(hunting): dump: 2024-03-29T11:27:06.140 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:06.140 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:06.141 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:06.141 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:06.141 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:06.141 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:06.141 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:06.141 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:06.141 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:06.141 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:06.141 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.150+0000 7fd4297fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:06.141 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.150+0000 7fd4297fa640 1 -- 172.21.15.86:0/52323815 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1713856139 0 0) 0x7fd42c06ec40 con 0x7fd42c0a9d70 2024-03-29T11:27:06.141 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.150+0000 7fd4297fa640 1 -- 172.21.15.86:0/52323815 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd418004eb0 con 0x7fd42c0a9d70 2024-03-29T11:27:06.141 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.150+0000 7fd4297fa640 1 -- 172.21.15.86:0/52323815 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fd424001b10 con 0x7fd42c0a9d70 2024-03-29T11:27:06.141 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.150+0000 7fd4297fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:06.141 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.150+0000 7fd4297fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:06.142 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.150+0000 7fd4297fa640 10 monclient(hunting): dump: 2024-03-29T11:27:06.142 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:06.142 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:06.142 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:06.142 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:06.142 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:06.142 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:06.142 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:06.142 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:06.142 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:06.142 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:06.142 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.150+0000 7fd4297fa640 10 client.?.objecter ms_handle_connect 0x7fd42c1447b0 2024-03-29T11:27:06.142 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.151+0000 7fd4297fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:06.142 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.151+0000 7fd4297fa640 1 -- 172.21.15.86:0/52323815 <== mon.0 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 901645470 0 0) 0x7fd418004eb0 con 0x7fd42c0a9d70 2024-03-29T11:27:06.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.151+0000 7fd4297fa640 1 -- 172.21.15.86:0/52323815 >> v1:172.21.15.86:6790/0 conn(0x7fd42c1457f0 legacy=0x7fd42c141ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:06.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.151+0000 7fd4297fa640 1 -- 172.21.15.86:0/52323815 >> v1:172.21.15.196:6789/0 conn(0x7fd42c1447b0 legacy=0x7fd42c1e63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:06.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.151+0000 7fd4297fa640 10 monclient: _finish_hunting 0 2024-03-29T11:27:06.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.151+0000 7fd4297fa640 1 monclient: found mon.a 2024-03-29T11:27:06.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.151+0000 7fd4297fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:06.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.151+0000 7fd4297fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:27:06.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.151+0000 7fd4297fa640 1 -- 172.21.15.86:0/52323815 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd42c06ef20 con 0x7fd42c0a9d70 2024-03-29T11:27:06.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.151+0000 7fd4297fa640 10 monclient: _finish_auth 0 2024-03-29T11:27:06.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.151+0000 7fd4297fa640 10 monclient: _check_auth_tickets 2024-03-29T11:27:06.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.151+0000 7fd4297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:06.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.151+0000 7fd4297fa640 1 -- 172.21.15.86:0/52323815 <== mon.0 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd424002020 con 0x7fd42c0a9d70 2024-03-29T11:27:06.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.151+0000 7fd4297fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:06.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.151+0000 7fd4297fa640 1 -- 172.21.15.86:0/52323815 <== mon.0 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fd424003b10 con 0x7fd42c0a9d70 2024-03-29T11:27:06.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.151+0000 7fd4297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:06.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.151+0000 7fd4297fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:06.144 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.151+0000 7fd4297fa640 10 monclient: dump: 2024-03-29T11:27:06.144 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:06.144 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:06.144 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:06.144 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:06.144 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:06.144 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:06.144 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:06.144 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:06.144 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:06.144 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:06.144 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.152+0000 7fd432f2b640 5 monclient: authenticate success, global_id 4440 2024-03-29T11:27:06.144 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.152+0000 7fd432f2b640 10 monclient: _renew_subs 2024-03-29T11:27:06.144 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.152+0000 7fd432f2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:27:06.144 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.152+0000 7fd432f2b640 1 -- 172.21.15.86:0/52323815 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd42c071ff0 con 0x7fd42c0a9d70 2024-03-29T11:27:06.144 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.152+0000 7fd42a7fc640 4 set_mon_vals no callback set 2024-03-29T11:27:06.145 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.152+0000 7fd432f2b640 10 client.4440.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:06.145 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.152+0000 7fd432f2b640 10 monclient: _renew_subs 2024-03-29T11:27:06.145 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.152+0000 7fd432f2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:27:06.145 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.152+0000 7fd432f2b640 1 -- 172.21.15.86:0/52323815 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd42c071c10 con 0x7fd42c0a9d70 2024-03-29T11:27:06.145 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.152+0000 7fd432f2b640 1 librados: init done 2024-03-29T11:27:06.146 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.154+0000 7fd432f2b640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:06.146 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.154+0000 7fd432f2b640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:06.146 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.154+0000 7fd432f2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:27:06.147 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.154+0000 7fd432f2b640 1 -- 172.21.15.86:0/52323815 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd42c075f10 con 0x7fd42c0a9d70 2024-03-29T11:27:06.147 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.154+0000 7fd4297fa640 1 -- 172.21.15.86:0/52323815 <== mon.0 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7fd424019ae0 con 0x7fd42c0a9d70 2024-03-29T11:27:06.147 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.155+0000 7fd4297fa640 1 -- 172.21.15.86:0/52323815 <== mon.0 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7fd424083b00 con 0x7fd42c0a9d70 2024-03-29T11:27:06.147 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.155+0000 7fd4297fa640 10 client.4440.objecter ms_dispatch 0x7fd42c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:06.147 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.155+0000 7fd4297fa640 3 client.4440.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:06.147 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.155+0000 7fd42bfff640 10 monclient: get_auth_request con 0x7fd41806de50 auth_method 0 2024-03-29T11:27:06.147 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.155+0000 7fd4297fa640 3 client.4440.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:06.147 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.155+0000 7fd4297fa640 20 client.4440.objecter dump_active .. 0 homeless 2024-03-29T11:27:06.147 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.156+0000 7fd4297fa640 10 client.4440.objecter ms_handle_connect 0x7fd41806de50 2024-03-29T11:27:06.152 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.163+0000 7fd4297fa640 1 -- 172.21.15.86:0/52323815 <== mon.0 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd42404f210 con 0x7fd42c0a9d70 2024-03-29T11:27:06.152 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.163+0000 7fd4297fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:06.152 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.163+0000 7fd4297fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:06.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.495+0000 7fd432f2b640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:06.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.495+0000 7fd432f2b640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:06.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.495+0000 7fd432f2b640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:27:06.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.495+0000 7fd432f2b640 1 -- 172.21.15.86:0/52323815 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd42c0b3680 con 0x7fd42c0a9d70 2024-03-29T11:27:06.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.497+0000 7fd4297fa640 1 -- 172.21.15.86:0/52323815 <== mon.0 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7fd424051310 con 0x7fd42c0a9d70 2024-03-29T11:27:06.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.497+0000 7fd4297fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:06.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.497+0000 7fd4297fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:06.486 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:27:06.486 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:06.489 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.501+0000 7fd432f2b640 10 librados: watch_flush enter 2024-03-29T11:27:06.489 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.501+0000 7fd432f2b640 10 librados: watch_flush exit 2024-03-29T11:27:06.489 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.501+0000 7fd432f2b640 20 client.4440.objecter shutdown clearing up homeless session... 2024-03-29T11:27:06.489 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.501+0000 7fd432f2b640 10 client.4440.objecter successfully canceled tick 2024-03-29T11:27:06.489 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.502+0000 7fd432f2b640 1 -- 172.21.15.86:0/52323815 >> v1:172.21.15.196:6816/588110318 conn(0x7fd41806de50 legacy=0x7fd418070310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:06.489 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.502+0000 7fd432f2b640 10 monclient: shutdown 2024-03-29T11:27:06.489 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.502+0000 7fd432f2b640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:06.489 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.502+0000 7fd432f2b640 1 -- 172.21.15.86:0/52323815 >> v1:172.21.15.86:6789/0 conn(0x7fd42c0a9d70 legacy=0x7fd42c1424a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:06.490 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.502+0000 7fd432f2b640 1 -- 172.21.15.86:0/52323815 shutdown_connections 2024-03-29T11:27:06.490 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.502+0000 7fd432f2b640 1 -- 172.21.15.86:0/52323815 >> 172.21.15.86:0/52323815 conn(0x7fd42c066a00 msgr2=0x7fd42c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:06.490 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.502+0000 7fd432f2b640 1 -- 172.21.15.86:0/52323815 shutdown_connections 2024-03-29T11:27:06.490 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.502+0000 7fd432f2b640 1 -- 172.21.15.86:0/52323815 wait complete. 2024-03-29T11:27:06.490 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:06.502+0000 7fd432f2b640 1 librados: shutdown 2024-03-29T11:27:06.506 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:13.314 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:13.463 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.476+0000 7feedf0d6640 10 monclient: get_monmap_and_config 2024-03-29T11:27:13.464 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.476+0000 7feedf0d6640 10 monclient: build_initial_monmap 2024-03-29T11:27:13.464 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.476+0000 7feedf0d6640 10 monclient: monmap: 2024-03-29T11:27:13.464 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:27:13.464 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:13.464 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:27:13.477307+0000 2024-03-29T11:27:13.464 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:27:13.477307+0000 2024-03-29T11:27:13.464 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:13.464 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:13.464 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:27:13.464 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:27:13.464 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:27:13.464 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:13.465 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.477+0000 7feedf0d6640 1 Processor -- start 2024-03-29T11:27:13.465 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.477+0000 7feedf0d6640 1 -- start start 2024-03-29T11:27:13.465 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.477+0000 7feedf0d6640 10 monclient: init 2024-03-29T11:27:13.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.478+0000 7feedf0d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:13.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.478+0000 7feedf0d6640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:13.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.478+0000 7feedf0d6640 10 monclient(hunting): picked mon.noname-a con 0x7feed806c250 addr v1:172.21.15.86:6789/0 2024-03-29T11:27:13.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.478+0000 7feedf0d6640 10 monclient(hunting): picked mon.noname-b con 0x7feed80ae650 addr v1:172.21.15.86:6790/0 2024-03-29T11:27:13.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.478+0000 7feedf0d6640 10 monclient(hunting): picked mon.noname-c con 0x7feed813f8c0 addr v1:172.21.15.196:6789/0 2024-03-29T11:27:13.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.478+0000 7feedf0d6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feed80adf50 con 0x7feed806c250 2024-03-29T11:27:13.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.478+0000 7feedf0d6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feed80ae230 con 0x7feed813f8c0 2024-03-29T11:27:13.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.478+0000 7feedf0d6640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feed813fd50 con 0x7feed80ae650 2024-03-29T11:27:13.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.478+0000 7feedf0d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:13.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.478+0000 7feedf0d6640 10 monclient(hunting): authenticate will time out at 1257.796539s 2024-03-29T11:27:13.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.478+0000 7feed7fff640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7feed80ae650 0x7feed813f1b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:34734/0 (socket says 172.21.15.86:34734) 2024-03-29T11:27:13.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.478+0000 7feed7fff640 1 -- 172.21.15.86:0/1625578654 learned_addr learned my addr 172.21.15.86:0/1625578654 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:27:13.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feed77fe640 1 -- 172.21.15.86:0/1625578654 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 223939051 0 0) 0x7feed813fd50 con 0x7feed80ae650 2024-03-29T11:27:13.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feed77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:13.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feed77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:13.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feed77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:13.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feed77fe640 10 monclient(hunting): my global_id is 4502 2024-03-29T11:27:13.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feed77fe640 1 -- 172.21.15.86:0/1625578654 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feec0003520 con 0x7feed80ae650 2024-03-29T11:27:13.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feed77fe640 1 -- 172.21.15.86:0/1625578654 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2367946341 0 0) 0x7feed80adf50 con 0x7feed806c250 2024-03-29T11:27:13.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feed77fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:13.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feed77fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:13.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feed77fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:13.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feed77fe640 10 monclient(hunting): my global_id is 4443 2024-03-29T11:27:13.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feed77fe640 1 -- 172.21.15.86:0/1625578654 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feed813fd50 con 0x7feed806c250 2024-03-29T11:27:13.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feed77fe640 1 -- 172.21.15.86:0/1625578654 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3570018343 0 0) 0x7feec0003520 con 0x7feed80ae650 2024-03-29T11:27:13.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feed77fe640 1 -- 172.21.15.86:0/1625578654 >> v1:172.21.15.196:6789/0 conn(0x7feed813f8c0 legacy=0x7feed814db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:13.470 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feed77fe640 1 -- 172.21.15.86:0/1625578654 >> v1:172.21.15.86:6789/0 conn(0x7feed806c250 legacy=0x7feed81427b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:13.470 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feed77fe640 10 monclient: _finish_hunting 0 2024-03-29T11:27:13.470 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feed77fe640 1 monclient: found mon.noname-b 2024-03-29T11:27:13.470 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feed77fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:13.470 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feed77fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:27:13.470 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feed77fe640 1 -- 172.21.15.86:0/1625578654 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed806db20 con 0x7feed80ae650 2024-03-29T11:27:13.470 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feed77fe640 10 monclient: _finish_auth 0 2024-03-29T11:27:13.471 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feed77fe640 10 monclient: _check_auth_tickets 2024-03-29T11:27:13.471 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feed77fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:13.471 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feed77fe640 1 -- 172.21.15.86:0/1625578654 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7feecc001690 con 0x7feed80ae650 2024-03-29T11:27:13.471 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feed77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:13.471 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feed77fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:13.471 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feed77fe640 10 monclient: dump: 2024-03-29T11:27:13.471 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:13.471 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:13.471 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:13.471 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:13.471 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:13.471 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:13.471 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:13.471 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:13.471 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:13.472 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:13.472 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feedf0d6640 5 monclient: authenticate success, global_id 4502 2024-03-29T11:27:13.472 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feedf0d6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:13.472 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feed77fe640 1 -- 172.21.15.86:0/1625578654 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feecc0017f0 con 0x7feed80ae650 2024-03-29T11:27:13.472 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feed77fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:13.472 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feed77fe640 1 -- 172.21.15.86:0/1625578654 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7feecc001c00 con 0x7feed80ae650 2024-03-29T11:27:13.472 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feed77fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:13.472 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feed77fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:13.472 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feed77fe640 10 monclient: dump: 2024-03-29T11:27:13.472 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:13.472 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:13.472 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:13.472 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:13.472 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:13.472 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:13.473 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:13.473 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:13.473 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:13.473 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:13.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feedf0d6640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:13.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.479+0000 7feedf0d6640 4 set_mon_vals no callback set 2024-03-29T11:27:13.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.480+0000 7feedf0d6640 10 monclient: shutdown 2024-03-29T11:27:13.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.480+0000 7feedf0d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:13.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.480+0000 7feedf0d6640 1 -- 172.21.15.86:0/1625578654 >> v1:172.21.15.86:6790/0 conn(0x7feed80ae650 legacy=0x7feed813f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:13.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.481+0000 7feedf0d6640 1 -- 172.21.15.86:0/1625578654 shutdown_connections 2024-03-29T11:27:13.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.481+0000 7feedf0d6640 1 -- 172.21.15.86:0/1625578654 >> 172.21.15.86:0/1625578654 conn(0x7feed8067610 msgr2=0x7feed80b5a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:13.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.481+0000 7feedf0d6640 1 -- 172.21.15.86:0/1625578654 shutdown_connections 2024-03-29T11:27:13.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.481+0000 7feedf0d6640 1 -- 172.21.15.86:0/1625578654 wait complete. 2024-03-29T11:27:13.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.482+0000 7feedf0d6640 10 monclient: build_initial_monmap 2024-03-29T11:27:13.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.482+0000 7feedf0d6640 10 monclient: monmap: 2024-03-29T11:27:13.479 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:27:13.479 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:13.479 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:27:13.479 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:27:13.479 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:13.479 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:13.479 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:27:13.479 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:27:13.479 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:27:13.479 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:13.479 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.482+0000 7feedf0d6640 1 librados: starting msgr at 2024-03-29T11:27:13.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.482+0000 7feedf0d6640 1 librados: starting objecter 2024-03-29T11:27:13.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.482+0000 7feedf0d6640 1 Processor -- start 2024-03-29T11:27:13.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.483+0000 7feedf0d6640 1 -- start start 2024-03-29T11:27:13.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.483+0000 7feedf0d6640 1 librados: setting wanted keys 2024-03-29T11:27:13.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.483+0000 7feedf0d6640 1 librados: calling monclient init 2024-03-29T11:27:13.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.483+0000 7feedf0d6640 10 monclient: init 2024-03-29T11:27:13.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.484+0000 7feedf0d6640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:13.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.484+0000 7feedf0d6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:13.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.484+0000 7feedf0d6640 10 monclient(hunting): picked mon.noname-b con 0x7feed806c250 addr v1:172.21.15.196:6789/0 2024-03-29T11:27:13.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.484+0000 7feedf0d6640 10 monclient(hunting): picked mon.noname-a con 0x7feed80ae650 addr v1:172.21.15.86:6789/0 2024-03-29T11:27:13.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.484+0000 7feedf0d6640 10 monclient(hunting): picked mon.noname-c con 0x7feed813f8c0 addr v1:172.21.15.86:6790/0 2024-03-29T11:27:13.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.484+0000 7feedf0d6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feed8143e30 con 0x7feed80ae650 2024-03-29T11:27:13.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.484+0000 7feedf0d6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feed8071fc0 con 0x7feed806c250 2024-03-29T11:27:13.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.484+0000 7feedf0d6640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feed80724e0 con 0x7feed813f8c0 2024-03-29T11:27:13.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.484+0000 7feedf0d6640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:13.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.484+0000 7feedf0d6640 10 monclient(hunting): authenticate will time out at 1257.802206s 2024-03-29T11:27:13.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.484+0000 7feedd673640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7feed813f8c0 0x7feed81e2a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:35728/0 (socket says 172.21.15.86:35728) 2024-03-29T11:27:13.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.484+0000 7feedd673640 1 -- 172.21.15.86:0/590469123 learned_addr learned my addr 172.21.15.86:0/590469123 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:27:13.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.484+0000 7feed57fa640 10 client.?.objecter ms_handle_connect 0x7feed813f8c0 2024-03-29T11:27:13.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.484+0000 7feed57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:13.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.484+0000 7feed57fa640 1 -- 172.21.15.86:0/590469123 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3957909671 0 0) 0x7feed80724e0 con 0x7feed813f8c0 2024-03-29T11:27:13.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.484+0000 7feed57fa640 10 monclient(hunting): _negotiate 2024-03-29T11:27:13.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.484+0000 7feed57fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:13.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.484+0000 7feed57fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:13.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.484+0000 7feed57fa640 10 monclient(hunting): my global_id is 4505 2024-03-29T11:27:13.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.484+0000 7feed57fa640 1 -- 172.21.15.86:0/590469123 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feec4004eb0 con 0x7feed813f8c0 2024-03-29T11:27:13.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.485+0000 7feed57fa640 10 client.?.objecter ms_handle_connect 0x7feed80ae650 2024-03-29T11:27:13.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.485+0000 7feed57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:13.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.485+0000 7feed57fa640 1 -- 172.21.15.86:0/590469123 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 106008971 0 0) 0x7feec4004eb0 con 0x7feed813f8c0 2024-03-29T11:27:13.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.485+0000 7feed57fa640 1 -- 172.21.15.86:0/590469123 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feed80724e0 con 0x7feed813f8c0 2024-03-29T11:27:13.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.485+0000 7feed57fa640 1 -- 172.21.15.86:0/590469123 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7feec8003f60 con 0x7feed813f8c0 2024-03-29T11:27:13.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.485+0000 7feed57fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:13.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.485+0000 7feed57fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:13.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.485+0000 7feed57fa640 10 monclient(hunting): dump: 2024-03-29T11:27:13.483 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:13.483 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:13.483 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:13.483 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:13.484 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:13.484 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:13.484 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:13.484 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:13.484 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:13.484 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:13.484 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.485+0000 7feed57fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:13.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.485+0000 7feed57fa640 10 client.?.objecter ms_handle_connect 0x7feed806c250 2024-03-29T11:27:13.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.485+0000 7feed57fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:13.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.485+0000 7feed57fa640 1 -- 172.21.15.86:0/590469123 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 310307058 0 0) 0x7feed80724e0 con 0x7feed813f8c0 2024-03-29T11:27:13.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.485+0000 7feed57fa640 1 -- 172.21.15.86:0/590469123 >> v1:172.21.15.196:6789/0 conn(0x7feed806c250 legacy=0x7feed81da6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:13.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.485+0000 7feed57fa640 1 -- 172.21.15.86:0/590469123 >> v1:172.21.15.86:6789/0 conn(0x7feed80ae650 legacy=0x7feed8134aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:13.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.485+0000 7feed57fa640 10 monclient: _finish_hunting 0 2024-03-29T11:27:13.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.485+0000 7feed57fa640 1 monclient: found mon.c 2024-03-29T11:27:13.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.485+0000 7feed57fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:13.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.485+0000 7feed57fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:27:13.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.485+0000 7feed57fa640 1 -- 172.21.15.86:0/590469123 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed81dadf0 con 0x7feed813f8c0 2024-03-29T11:27:13.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.485+0000 7feed57fa640 10 monclient: _finish_auth 0 2024-03-29T11:27:13.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.485+0000 7feed57fa640 10 monclient: _check_auth_tickets 2024-03-29T11:27:13.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.485+0000 7feed57fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:13.487 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.485+0000 7feedf0d6640 5 monclient: authenticate success, global_id 4505 2024-03-29T11:27:13.487 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.485+0000 7feedf0d6640 10 monclient: _renew_subs 2024-03-29T11:27:13.487 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.485+0000 7feedf0d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:27:13.487 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.485+0000 7feedf0d6640 1 -- 172.21.15.86:0/590469123 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feed81db0a0 con 0x7feed813f8c0 2024-03-29T11:27:13.487 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.485+0000 7feedf0d6640 10 client.4505.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:13.487 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.485+0000 7feedf0d6640 10 monclient: _renew_subs 2024-03-29T11:27:13.487 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.485+0000 7feedf0d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:27:13.488 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.485+0000 7feedf0d6640 1 -- 172.21.15.86:0/590469123 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feed8071730 con 0x7feed813f8c0 2024-03-29T11:27:13.488 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.485+0000 7feedf0d6640 1 librados: init done 2024-03-29T11:27:13.488 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.486+0000 7feed57fa640 1 -- 172.21.15.86:0/590469123 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feec80041f0 con 0x7feed813f8c0 2024-03-29T11:27:13.488 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.486+0000 7feed57fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:13.488 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.486+0000 7feed57fa640 1 -- 172.21.15.86:0/590469123 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7feec80010f0 con 0x7feed813f8c0 2024-03-29T11:27:13.488 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.486+0000 7feed57fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:13.489 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.486+0000 7feed57fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:13.489 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.486+0000 7feed6ffd640 4 set_mon_vals no callback set 2024-03-29T11:27:13.489 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.486+0000 7feed57fa640 10 monclient: dump: 2024-03-29T11:27:13.489 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:13.489 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:13.489 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:13.489 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:13.490 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:13.490 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:13.490 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:13.490 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:13.490 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:13.490 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:13.490 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.487+0000 7feedf0d6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:13.491 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.487+0000 7feedf0d6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:13.491 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.487+0000 7feedf0d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:27:13.491 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.487+0000 7feedf0d6640 1 -- 172.21.15.86:0/590469123 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feeb8005080 con 0x7feed813f8c0 2024-03-29T11:27:13.491 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.488+0000 7feed57fa640 1 -- 172.21.15.86:0/590469123 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7feec8001350 con 0x7feed813f8c0 2024-03-29T11:27:13.491 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.488+0000 7feed57fa640 1 -- 172.21.15.86:0/590469123 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7feec8084a20 con 0x7feed813f8c0 2024-03-29T11:27:13.491 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.488+0000 7feed57fa640 10 client.4505.objecter ms_dispatch 0x7feed8000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:13.492 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.488+0000 7feed57fa640 3 client.4505.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:13.492 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.488+0000 7feed57fa640 3 client.4505.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:13.492 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.488+0000 7feed57fa640 20 client.4505.objecter dump_active .. 0 homeless 2024-03-29T11:27:13.492 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.489+0000 7feedce72640 10 monclient: get_auth_request con 0x7feec406dc90 auth_method 0 2024-03-29T11:27:13.492 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.489+0000 7feed57fa640 10 client.4505.objecter ms_handle_connect 0x7feec406dc90 2024-03-29T11:27:13.493 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.497+0000 7feed57fa640 1 -- 172.21.15.86:0/590469123 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feec8050300 con 0x7feed813f8c0 2024-03-29T11:27:13.493 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.498+0000 7feed57fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:13.493 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.498+0000 7feed57fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:13.828 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.839+0000 7feedf0d6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:13.828 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.839+0000 7feedf0d6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:13.828 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.839+0000 7feedf0d6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:27:13.828 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.839+0000 7feedf0d6640 1 -- 172.21.15.86:0/590469123 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feeb8005330 con 0x7feed813f8c0 2024-03-29T11:27:13.828 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.840+0000 7feed57fa640 1 -- 172.21.15.86:0/590469123 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7feec8052230 con 0x7feed813f8c0 2024-03-29T11:27:13.828 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.840+0000 7feed57fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:13.828 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.840+0000 7feed57fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:13.829 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:27:13.829 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:13.833 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.843+0000 7feedf0d6640 10 librados: watch_flush enter 2024-03-29T11:27:13.833 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.843+0000 7feedf0d6640 10 librados: watch_flush exit 2024-03-29T11:27:13.834 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.843+0000 7feedf0d6640 20 client.4505.objecter shutdown clearing up homeless session... 2024-03-29T11:27:13.834 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.843+0000 7feedf0d6640 10 client.4505.objecter successfully canceled tick 2024-03-29T11:27:13.834 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.843+0000 7feedf0d6640 1 -- 172.21.15.86:0/590469123 >> v1:172.21.15.196:6816/588110318 conn(0x7feec406dc90 legacy=0x7feec4070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:13.834 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.843+0000 7feedf0d6640 10 monclient: shutdown 2024-03-29T11:27:13.834 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.843+0000 7feedf0d6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:13.834 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.843+0000 7feedf0d6640 1 -- 172.21.15.86:0/590469123 >> v1:172.21.15.86:6790/0 conn(0x7feed813f8c0 legacy=0x7feed81e2a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:13.835 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.843+0000 7feedf0d6640 1 -- 172.21.15.86:0/590469123 shutdown_connections 2024-03-29T11:27:13.835 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.843+0000 7feedf0d6640 1 -- 172.21.15.86:0/590469123 >> 172.21.15.86:0/590469123 conn(0x7feed8067610 msgr2=0x7feed813ef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:13.835 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.844+0000 7feedf0d6640 1 -- 172.21.15.86:0/590469123 shutdown_connections 2024-03-29T11:27:13.835 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.844+0000 7feedf0d6640 1 -- 172.21.15.86:0/590469123 wait complete. 2024-03-29T11:27:13.835 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:13.844+0000 7feedf0d6640 1 librados: shutdown 2024-03-29T11:27:13.846 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:20.653 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:20.784 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.793+0000 7fd13e84e640 10 monclient: get_monmap_and_config 2024-03-29T11:27:20.785 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.793+0000 7fd13e84e640 10 monclient: build_initial_monmap 2024-03-29T11:27:20.785 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.794+0000 7fd13e84e640 10 monclient: monmap: 2024-03-29T11:27:20.785 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:27:20.785 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:20.785 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:27:20.794953+0000 2024-03-29T11:27:20.785 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:27:20.794953+0000 2024-03-29T11:27:20.785 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:20.785 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:20.785 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:27:20.785 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:27:20.785 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:27:20.785 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:20.785 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.795+0000 7fd13e84e640 1 Processor -- start 2024-03-29T11:27:20.786 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.795+0000 7fd13e84e640 1 -- start start 2024-03-29T11:27:20.786 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.795+0000 7fd13e84e640 10 monclient: init 2024-03-29T11:27:20.786 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.795+0000 7fd13e84e640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:20.786 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.795+0000 7fd13e84e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:20.786 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.795+0000 7fd13e84e640 10 monclient(hunting): picked mon.noname-c con 0x7fd1380a9ac0 addr v1:172.21.15.196:6789/0 2024-03-29T11:27:20.786 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.796+0000 7fd13e84e640 10 monclient(hunting): picked mon.noname-a con 0x7fd13813ee90 addr v1:172.21.15.86:6789/0 2024-03-29T11:27:20.786 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.796+0000 7fd13e84e640 10 monclient(hunting): picked mon.noname-b con 0x7fd1380af900 addr v1:172.21.15.86:6790/0 2024-03-29T11:27:20.786 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.796+0000 7fd13e84e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1380afd10 con 0x7fd13813ee90 2024-03-29T11:27:20.786 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.796+0000 7fd13e84e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1380af6f0 con 0x7fd1380a9ac0 2024-03-29T11:27:20.786 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.796+0000 7fd13e84e640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1380aac80 con 0x7fd1380af900 2024-03-29T11:27:20.786 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.796+0000 7fd13e84e640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:20.786 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.796+0000 7fd13e84e640 10 monclient(hunting): authenticate will time out at 1265.114082s 2024-03-29T11:27:20.786 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.796+0000 7fd13cdeb640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7fd1380af900 0x7fd138149b00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:41732/0 (socket says 172.21.15.86:41732) 2024-03-29T11:27:20.787 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.796+0000 7fd13cdeb640 1 -- 172.21.15.86:0/563731506 learned_addr learned my addr 172.21.15.86:0/563731506 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:27:20.787 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.796+0000 7fd136ffd640 1 -- 172.21.15.86:0/563731506 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1401185615 0 0) 0x7fd1380aac80 con 0x7fd1380af900 2024-03-29T11:27:20.787 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.796+0000 7fd136ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:20.787 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.796+0000 7fd136ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:20.787 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.796+0000 7fd136ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:20.787 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.796+0000 7fd136ffd640 10 monclient(hunting): my global_id is 4508 2024-03-29T11:27:20.787 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.796+0000 7fd136ffd640 1 -- 172.21.15.86:0/563731506 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd120003520 con 0x7fd1380af900 2024-03-29T11:27:20.787 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.796+0000 7fd136ffd640 1 -- 172.21.15.86:0/563731506 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 787856804 0 0) 0x7fd120003520 con 0x7fd1380af900 2024-03-29T11:27:20.787 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.796+0000 7fd136ffd640 1 -- 172.21.15.86:0/563731506 >> v1:172.21.15.196:6789/0 conn(0x7fd1380a9ac0 legacy=0x7fd1380adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:20.787 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.796+0000 7fd136ffd640 1 -- 172.21.15.86:0/563731506 >> v1:172.21.15.86:6789/0 conn(0x7fd13813ee90 legacy=0x7fd13813b5f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:20.788 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.799+0000 7fd136ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:27:20.788 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.799+0000 7fd136ffd640 1 monclient: found mon.noname-b 2024-03-29T11:27:20.788 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.799+0000 7fd136ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:20.788 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.799+0000 7fd136ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:27:20.788 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.799+0000 7fd136ffd640 1 -- 172.21.15.86:0/563731506 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1380aaf60 con 0x7fd1380af900 2024-03-29T11:27:20.789 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.799+0000 7fd136ffd640 10 monclient: _finish_auth 0 2024-03-29T11:27:20.789 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.799+0000 7fd136ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:27:20.789 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.799+0000 7fd136ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:20.789 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.799+0000 7fd136ffd640 1 -- 172.21.15.86:0/563731506 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fd128001690 con 0x7fd1380af900 2024-03-29T11:27:20.789 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.799+0000 7fd136ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:20.789 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.799+0000 7fd136ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:20.789 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.799+0000 7fd136ffd640 10 monclient: dump: 2024-03-29T11:27:20.789 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:20.789 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:20.789 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:20.789 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:20.789 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:20.789 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:20.789 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:20.789 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:20.790 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:20.790 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:20.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.799+0000 7fd13e84e640 5 monclient: authenticate success, global_id 4508 2024-03-29T11:27:20.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.799+0000 7fd13e84e640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:20.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.800+0000 7fd136ffd640 1 -- 172.21.15.86:0/563731506 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd1280018f0 con 0x7fd1380af900 2024-03-29T11:27:20.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.800+0000 7fd136ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:20.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.800+0000 7fd136ffd640 1 -- 172.21.15.86:0/563731506 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fd128001d40 con 0x7fd1380af900 2024-03-29T11:27:20.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.800+0000 7fd136ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:20.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.800+0000 7fd136ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:20.790 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.800+0000 7fd136ffd640 10 monclient: dump: 2024-03-29T11:27:20.790 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:20.790 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:20.790 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:20.791 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:20.791 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:20.791 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:20.791 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:20.791 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:20.791 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:20.791 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:20.791 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.800+0000 7fd13e84e640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:20.791 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.800+0000 7fd13e84e640 4 set_mon_vals no callback set 2024-03-29T11:27:20.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.801+0000 7fd13e84e640 10 monclient: shutdown 2024-03-29T11:27:20.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.801+0000 7fd13e84e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:20.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.801+0000 7fd13e84e640 1 -- 172.21.15.86:0/563731506 >> v1:172.21.15.86:6790/0 conn(0x7fd1380af900 legacy=0x7fd138149b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:20.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.801+0000 7fd13e84e640 1 -- 172.21.15.86:0/563731506 shutdown_connections 2024-03-29T11:27:20.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.801+0000 7fd13e84e640 1 -- 172.21.15.86:0/563731506 >> 172.21.15.86:0/563731506 conn(0x7fd1380b35e0 msgr2=0x7fd1380b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:20.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.801+0000 7fd13e84e640 1 -- 172.21.15.86:0/563731506 shutdown_connections 2024-03-29T11:27:20.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.801+0000 7fd13e84e640 1 -- 172.21.15.86:0/563731506 wait complete. 2024-03-29T11:27:20.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.802+0000 7fd13e84e640 10 monclient: build_initial_monmap 2024-03-29T11:27:20.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.802+0000 7fd13e84e640 10 monclient: monmap: 2024-03-29T11:27:20.796 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:27:20.797 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:20.797 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:27:20.797 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:27:20.797 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:20.797 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:20.797 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:27:20.797 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:27:20.797 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:27:20.797 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:20.797 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.802+0000 7fd13e84e640 1 librados: starting msgr at 2024-03-29T11:27:20.797 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.802+0000 7fd13e84e640 1 librados: starting objecter 2024-03-29T11:27:20.797 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.803+0000 7fd13e84e640 1 Processor -- start 2024-03-29T11:27:20.797 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.803+0000 7fd13e84e640 1 -- start start 2024-03-29T11:27:20.797 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.803+0000 7fd13e84e640 1 librados: setting wanted keys 2024-03-29T11:27:20.798 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.803+0000 7fd13e84e640 1 librados: calling monclient init 2024-03-29T11:27:20.798 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.803+0000 7fd13e84e640 10 monclient: init 2024-03-29T11:27:20.798 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.803+0000 7fd13e84e640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:20.798 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.803+0000 7fd13e84e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:20.798 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.803+0000 7fd13e84e640 10 monclient(hunting): picked mon.noname-b con 0x7fd1380a9ac0 addr v1:172.21.15.196:6789/0 2024-03-29T11:27:20.798 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.803+0000 7fd13e84e640 10 monclient(hunting): picked mon.noname-a con 0x7fd1380af900 addr v1:172.21.15.86:6789/0 2024-03-29T11:27:20.798 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.803+0000 7fd13e84e640 10 monclient(hunting): picked mon.noname-c con 0x7fd13813ee90 addr v1:172.21.15.86:6790/0 2024-03-29T11:27:20.798 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.803+0000 7fd13e84e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1380724c0 con 0x7fd1380af900 2024-03-29T11:27:20.798 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.803+0000 7fd13e84e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1380716b0 con 0x7fd1380a9ac0 2024-03-29T11:27:20.799 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.803+0000 7fd13e84e640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd138071990 con 0x7fd13813ee90 2024-03-29T11:27:20.799 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.803+0000 7fd13e84e640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:20.799 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.803+0000 7fd13e84e640 10 monclient(hunting): authenticate will time out at 1265.121902s 2024-03-29T11:27:20.799 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.804+0000 7fd13cdeb640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7fd13813ee90 0x7fd138071db0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:44392/0 (socket says 172.21.15.86:44392) 2024-03-29T11:27:20.799 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.804+0000 7fd13cdeb640 1 -- 172.21.15.86:0/3150392852 learned_addr learned my addr 172.21.15.86:0/3150392852 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:27:20.799 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.804+0000 7fd134ff9640 10 client.?.objecter ms_handle_connect 0x7fd13813ee90 2024-03-29T11:27:20.799 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.804+0000 7fd134ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:20.799 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.804+0000 7fd134ff9640 1 -- 172.21.15.86:0/3150392852 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2774914049 0 0) 0x7fd138071990 con 0x7fd13813ee90 2024-03-29T11:27:20.799 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.804+0000 7fd134ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:27:20.799 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.804+0000 7fd134ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:20.799 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.804+0000 7fd134ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:20.800 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.804+0000 7fd134ff9640 10 monclient(hunting): my global_id is 4511 2024-03-29T11:27:20.800 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.804+0000 7fd134ff9640 1 -- 172.21.15.86:0/3150392852 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd124004eb0 con 0x7fd13813ee90 2024-03-29T11:27:20.800 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.805+0000 7fd134ff9640 10 client.?.objecter ms_handle_connect 0x7fd1380a9ac0 2024-03-29T11:27:20.800 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.805+0000 7fd134ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:20.800 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.805+0000 7fd134ff9640 1 -- 172.21.15.86:0/3150392852 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2621428013 0 0) 0x7fd124004eb0 con 0x7fd13813ee90 2024-03-29T11:27:20.800 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.805+0000 7fd134ff9640 1 -- 172.21.15.86:0/3150392852 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd138071990 con 0x7fd13813ee90 2024-03-29T11:27:20.800 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.805+0000 7fd134ff9640 1 -- 172.21.15.86:0/3150392852 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fd1280014b0 con 0x7fd13813ee90 2024-03-29T11:27:20.800 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.805+0000 7fd134ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:20.800 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.805+0000 7fd134ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:20.800 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.805+0000 7fd134ff9640 10 monclient(hunting): dump: 2024-03-29T11:27:20.800 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:20.800 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:20.800 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:20.800 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:20.800 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:20.801 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:20.801 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:20.801 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:20.801 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:20.801 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:20.801 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.805+0000 7fd134ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:20.801 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.805+0000 7fd134ff9640 1 -- 172.21.15.86:0/3150392852 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2345949071 0 0) 0x7fd138071990 con 0x7fd13813ee90 2024-03-29T11:27:20.801 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.805+0000 7fd134ff9640 1 -- 172.21.15.86:0/3150392852 >> v1:172.21.15.196:6789/0 conn(0x7fd1380a9ac0 legacy=0x7fd13806fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:20.801 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.806+0000 7fd134ff9640 1 -- 172.21.15.86:0/3150392852 >> v1:172.21.15.86:6789/0 conn(0x7fd1380af900 legacy=0x7fd13806f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:20.801 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.806+0000 7fd134ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:27:20.801 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.806+0000 7fd134ff9640 1 monclient: found mon.c 2024-03-29T11:27:20.801 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.806+0000 7fd134ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:20.801 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.806+0000 7fd134ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:27:20.801 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.806+0000 7fd134ff9640 1 -- 172.21.15.86:0/3150392852 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd13813baa0 con 0x7fd13813ee90 2024-03-29T11:27:20.801 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.806+0000 7fd134ff9640 10 monclient: _finish_auth 0 2024-03-29T11:27:20.802 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.806+0000 7fd134ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:27:20.802 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.806+0000 7fd134ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:20.802 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.806+0000 7fd13e84e640 5 monclient: authenticate success, global_id 4511 2024-03-29T11:27:20.802 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.807+0000 7fd13e84e640 10 monclient: _renew_subs 2024-03-29T11:27:20.802 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.807+0000 7fd13e84e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:27:20.802 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.807+0000 7fd13e84e640 1 -- 172.21.15.86:0/3150392852 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd13812cd50 con 0x7fd13813ee90 2024-03-29T11:27:20.802 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.807+0000 7fd13e84e640 10 client.4511.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:20.802 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.807+0000 7fd13e84e640 10 monclient: _renew_subs 2024-03-29T11:27:20.802 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.807+0000 7fd13e84e640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:27:20.802 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.807+0000 7fd13e84e640 1 -- 172.21.15.86:0/3150392852 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1381e0d60 con 0x7fd13813ee90 2024-03-29T11:27:20.802 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.807+0000 7fd13e84e640 1 librados: init done 2024-03-29T11:27:20.802 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.807+0000 7fd134ff9640 1 -- 172.21.15.86:0/3150392852 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd128004c70 con 0x7fd13813ee90 2024-03-29T11:27:20.803 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.807+0000 7fd134ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:20.803 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.807+0000 7fd134ff9640 1 -- 172.21.15.86:0/3150392852 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fd128004f80 con 0x7fd13813ee90 2024-03-29T11:27:20.803 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.807+0000 7fd134ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:20.803 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.807+0000 7fd134ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:20.803 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.807+0000 7fd134ff9640 10 monclient: dump: 2024-03-29T11:27:20.803 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:20.803 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:20.803 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:20.803 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:20.803 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:20.803 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:20.803 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:20.803 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:20.804 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:20.804 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:20.804 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.807+0000 7fd135ffb640 4 set_mon_vals no callback set 2024-03-29T11:27:20.804 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.808+0000 7fd11e7fc640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:20.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.808+0000 7fd11e7fc640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:20.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.809+0000 7fd11e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:27:20.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.809+0000 7fd134ff9640 1 -- 172.21.15.86:0/3150392852 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7fd12801af50 con 0x7fd13813ee90 2024-03-29T11:27:20.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.809+0000 7fd11e7fc640 1 -- 172.21.15.86:0/3150392852 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd110005080 con 0x7fd13813ee90 2024-03-29T11:27:20.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.809+0000 7fd134ff9640 1 -- 172.21.15.86:0/3150392852 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7fd128084ca0 con 0x7fd13813ee90 2024-03-29T11:27:20.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.810+0000 7fd134ff9640 10 client.4511.objecter ms_dispatch 0x7fd138000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:20.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.810+0000 7fd134ff9640 3 client.4511.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:20.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.810+0000 7fd134ff9640 3 client.4511.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:20.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.810+0000 7fd134ff9640 20 client.4511.objecter dump_active .. 0 homeless 2024-03-29T11:27:20.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.810+0000 7fd137fff640 10 monclient: get_auth_request con 0x7fd12406ddd0 auth_method 0 2024-03-29T11:27:20.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.810+0000 7fd134ff9640 10 client.4511.objecter ms_handle_connect 0x7fd12406ddd0 2024-03-29T11:27:20.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.817+0000 7fd134ff9640 1 -- 172.21.15.86:0/3150392852 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd128050580 con 0x7fd13813ee90 2024-03-29T11:27:20.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.817+0000 7fd134ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:20.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:20.817+0000 7fd134ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:21.120 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:21.132+0000 7fd11e7fc640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:21.120 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:21.132+0000 7fd11e7fc640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:21.120 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:21.132+0000 7fd11e7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:27:21.121 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:21.132+0000 7fd11e7fc640 1 -- 172.21.15.86:0/3150392852 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd110005330 con 0x7fd13813ee90 2024-03-29T11:27:21.121 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:21.133+0000 7fd134ff9640 1 -- 172.21.15.86:0/3150392852 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7fd1280524b0 con 0x7fd13813ee90 2024-03-29T11:27:21.121 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:21.133+0000 7fd134ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:21.121 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:21.133+0000 7fd134ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:21.122 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:27:21.122 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:21.125 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:21.137+0000 7fd11e7fc640 10 librados: watch_flush enter 2024-03-29T11:27:21.125 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:21.137+0000 7fd11e7fc640 10 librados: watch_flush exit 2024-03-29T11:27:21.125 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:21.137+0000 7fd11e7fc640 20 client.4511.objecter shutdown clearing up homeless session... 2024-03-29T11:27:21.125 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:21.137+0000 7fd11e7fc640 10 client.4511.objecter successfully canceled tick 2024-03-29T11:27:21.129 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:21.140+0000 7fd11e7fc640 1 -- 172.21.15.86:0/3150392852 >> v1:172.21.15.196:6816/588110318 conn(0x7fd12406ddd0 legacy=0x7fd124070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:21.129 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:21.140+0000 7fd11e7fc640 10 monclient: shutdown 2024-03-29T11:27:21.129 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:21.140+0000 7fd11e7fc640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:21.129 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:21.140+0000 7fd11e7fc640 1 -- 172.21.15.86:0/3150392852 >> v1:172.21.15.86:6790/0 conn(0x7fd13813ee90 legacy=0x7fd138071db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:21.129 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:21.141+0000 7fd11e7fc640 1 -- 172.21.15.86:0/3150392852 shutdown_connections 2024-03-29T11:27:21.129 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:21.141+0000 7fd11e7fc640 1 -- 172.21.15.86:0/3150392852 >> 172.21.15.86:0/3150392852 conn(0x7fd1380b35e0 msgr2=0x7fd1380adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:21.129 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:21.141+0000 7fd11e7fc640 1 -- 172.21.15.86:0/3150392852 shutdown_connections 2024-03-29T11:27:21.129 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:21.141+0000 7fd11e7fc640 1 -- 172.21.15.86:0/3150392852 wait complete. 2024-03-29T11:27:21.130 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:21.141+0000 7fd11e7fc640 1 librados: shutdown 2024-03-29T11:27:21.147 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:27.954 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:28.087 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.099+0000 7f6f35732640 10 monclient: get_monmap_and_config 2024-03-29T11:27:28.087 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.099+0000 7f6f35732640 10 monclient: build_initial_monmap 2024-03-29T11:27:28.087 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.099+0000 7f6f35732640 10 monclient: monmap: 2024-03-29T11:27:28.087 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:27:28.088 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:28.088 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:27:28.100152+0000 2024-03-29T11:27:28.088 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:27:28.100152+0000 2024-03-29T11:27:28.088 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:28.088 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:28.088 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:27:28.088 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:27:28.088 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:27:28.088 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:28.088 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.099+0000 7f6f35732640 1 Processor -- start 2024-03-29T11:27:28.088 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.099+0000 7f6f35732640 1 -- start start 2024-03-29T11:27:28.088 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.099+0000 7f6f35732640 10 monclient: init 2024-03-29T11:27:28.089 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.100+0000 7f6f35732640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:28.089 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.100+0000 7f6f35732640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:28.089 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.100+0000 7f6f35732640 10 monclient(hunting): picked mon.noname-c con 0x7f6f301457f0 addr v1:172.21.15.196:6789/0 2024-03-29T11:27:28.089 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.100+0000 7f6f35732640 10 monclient(hunting): picked mon.noname-a con 0x7f6f3013f650 addr v1:172.21.15.86:6789/0 2024-03-29T11:27:28.089 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.100+0000 7f6f35732640 10 monclient(hunting): picked mon.noname-b con 0x7f6f300a9d70 addr v1:172.21.15.86:6790/0 2024-03-29T11:27:28.089 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.100+0000 7f6f35732640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f300aaef0 con 0x7f6f3013f650 2024-03-29T11:27:28.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.100+0000 7f6f35732640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f30145bd0 con 0x7f6f301457f0 2024-03-29T11:27:28.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.100+0000 7f6f35732640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f30145eb0 con 0x7f6f300a9d70 2024-03-29T11:27:28.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.100+0000 7f6f35732640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:28.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.100+0000 7f6f35732640 10 monclient(hunting): authenticate will time out at 1272.418540s 2024-03-29T11:27:28.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.100+0000 7f6f2effd640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f6f301457f0 0x7f6f3013ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:35594/0 (socket says 172.21.15.86:35594) 2024-03-29T11:27:28.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.100+0000 7f6f2effd640 1 -- 172.21.15.86:0/836664411 learned_addr learned my addr 172.21.15.86:0/836664411 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:27:28.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.101+0000 7f6f2dffb640 1 -- 172.21.15.86:0/836664411 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3054698485 0 0) 0x7f6f30145bd0 con 0x7f6f301457f0 2024-03-29T11:27:28.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.101+0000 7f6f2dffb640 10 monclient(hunting): _negotiate 2024-03-29T11:27:28.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.101+0000 7f6f2dffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:28.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.101+0000 7f6f2dffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:28.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.101+0000 7f6f2dffb640 10 monclient(hunting): my global_id is 4438 2024-03-29T11:27:28.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.101+0000 7f6f2dffb640 1 -- 172.21.15.86:0/836664411 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f18003520 con 0x7f6f301457f0 2024-03-29T11:27:28.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.102+0000 7f6f2dffb640 1 -- 172.21.15.86:0/836664411 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1438894203 0 0) 0x7f6f18003520 con 0x7f6f301457f0 2024-03-29T11:27:28.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.102+0000 7f6f2dffb640 1 -- 172.21.15.86:0/836664411 >> v1:172.21.15.86:6790/0 conn(0x7f6f300a9d70 legacy=0x7f6f3014d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:28.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.102+0000 7f6f2dffb640 1 -- 172.21.15.86:0/836664411 >> v1:172.21.15.86:6789/0 conn(0x7f6f3013f650 legacy=0x7f6f300b2620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:28.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.104+0000 7f6f2dffb640 10 monclient: _finish_hunting 0 2024-03-29T11:27:28.093 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.104+0000 7f6f2dffb640 1 monclient: found mon.noname-c 2024-03-29T11:27:28.093 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.104+0000 7f6f2dffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:28.093 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.104+0000 7f6f2dffb640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T11:27:28.095 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.104+0000 7f6f2dffb640 1 -- 172.21.15.86:0/836664411 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f30146340 con 0x7f6f301457f0 2024-03-29T11:27:28.095 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.105+0000 7f6f2dffb640 10 monclient: _finish_auth 0 2024-03-29T11:27:28.095 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.105+0000 7f6f2dffb640 10 monclient: _check_auth_tickets 2024-03-29T11:27:28.095 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.105+0000 7f6f2dffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:28.095 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.105+0000 7f6f2dffb640 1 -- 172.21.15.86:0/836664411 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f6f20001690 con 0x7f6f301457f0 2024-03-29T11:27:28.095 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.105+0000 7f6f2dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:28.096 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.105+0000 7f6f2dffb640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:28.096 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.105+0000 7f6f2dffb640 10 monclient: dump: 2024-03-29T11:27:28.096 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:28.096 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:28.096 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:28.096 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:28.096 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:28.096 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:28.096 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:28.096 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:28.096 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:28.096 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:28.096 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.105+0000 7f6f35732640 5 monclient: authenticate success, global_id 4438 2024-03-29T11:27:28.096 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.105+0000 7f6f35732640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:28.096 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.105+0000 7f6f2dffb640 1 -- 172.21.15.86:0/836664411 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f20001860 con 0x7f6f301457f0 2024-03-29T11:27:28.097 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.105+0000 7f6f2dffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:28.097 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.105+0000 7f6f2dffb640 1 -- 172.21.15.86:0/836664411 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f6f20001cb0 con 0x7f6f301457f0 2024-03-29T11:27:28.097 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.105+0000 7f6f35732640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:28.097 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.105+0000 7f6f35732640 4 set_mon_vals no callback set 2024-03-29T11:27:28.097 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.105+0000 7f6f2dffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:28.097 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.105+0000 7f6f2dffb640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:28.097 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.105+0000 7f6f2dffb640 10 monclient: dump: 2024-03-29T11:27:28.097 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:28.097 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:28.097 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:28.097 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:28.097 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:28.097 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:28.097 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:28.098 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:28.098 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:28.098 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:28.098 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.106+0000 7f6f35732640 10 monclient: shutdown 2024-03-29T11:27:28.098 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.106+0000 7f6f35732640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:28.098 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.106+0000 7f6f35732640 1 -- 172.21.15.86:0/836664411 >> v1:172.21.15.196:6789/0 conn(0x7f6f301457f0 legacy=0x7f6f3013ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:28.098 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.106+0000 7f6f35732640 1 -- 172.21.15.86:0/836664411 shutdown_connections 2024-03-29T11:27:28.098 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.106+0000 7f6f35732640 1 -- 172.21.15.86:0/836664411 >> 172.21.15.86:0/836664411 conn(0x7f6f30066a00 msgr2=0x7f6f30066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:28.098 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.106+0000 7f6f35732640 1 -- 172.21.15.86:0/836664411 shutdown_connections 2024-03-29T11:27:28.098 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.106+0000 7f6f35732640 1 -- 172.21.15.86:0/836664411 wait complete. 2024-03-29T11:27:28.106 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.107+0000 7f6f35732640 10 monclient: build_initial_monmap 2024-03-29T11:27:28.106 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.107+0000 7f6f35732640 10 monclient: monmap: 2024-03-29T11:27:28.106 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:27:28.106 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:28.106 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:27:28.106 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:27:28.107 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:28.107 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:28.107 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:27:28.107 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:27:28.107 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:27:28.107 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:28.107 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.108+0000 7f6f35732640 1 librados: starting msgr at 2024-03-29T11:27:28.107 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.108+0000 7f6f35732640 1 librados: starting objecter 2024-03-29T11:27:28.107 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.108+0000 7f6f35732640 1 Processor -- start 2024-03-29T11:27:28.107 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.108+0000 7f6f35732640 1 -- start start 2024-03-29T11:27:28.107 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.108+0000 7f6f35732640 1 librados: setting wanted keys 2024-03-29T11:27:28.107 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.108+0000 7f6f35732640 1 librados: calling monclient init 2024-03-29T11:27:28.107 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.108+0000 7f6f35732640 10 monclient: init 2024-03-29T11:27:28.107 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.109+0000 7f6f35732640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:28.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.109+0000 7f6f35732640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:28.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.109+0000 7f6f35732640 10 monclient(hunting): picked mon.noname-b con 0x7f6f300a9d70 addr v1:172.21.15.196:6789/0 2024-03-29T11:27:28.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.109+0000 7f6f2effd640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f6f300a9d70 0x7f6f300ab600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:35954/0 (socket says 172.21.15.86:35954) 2024-03-29T11:27:28.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.111+0000 7f6f35732640 10 monclient(hunting): picked mon.noname-a con 0x7f6f3013f650 addr v1:172.21.15.86:6789/0 2024-03-29T11:27:28.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.111+0000 7f6f35732640 10 monclient(hunting): picked mon.noname-c con 0x7f6f301457f0 addr v1:172.21.15.86:6790/0 2024-03-29T11:27:28.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.111+0000 7f6f35732640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f301db340 con 0x7f6f3013f650 2024-03-29T11:27:28.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.111+0000 7f6f35732640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f301387a0 con 0x7f6f300a9d70 2024-03-29T11:27:28.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.111+0000 7f6f35732640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f300ad7a0 con 0x7f6f301457f0 2024-03-29T11:27:28.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.111+0000 7f6f35732640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:28.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.111+0000 7f6f35732640 10 monclient(hunting): authenticate will time out at 1272.429346s 2024-03-29T11:27:28.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.111+0000 7f6f2effd640 1 -- 172.21.15.86:0/1236857684 learned_addr learned my addr 172.21.15.86:0/1236857684 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:27:28.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.111+0000 7f6f177fe640 10 client.?.objecter ms_handle_connect 0x7f6f300a9d70 2024-03-29T11:27:28.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.111+0000 7f6f177fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:28.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.111+0000 7f6f177fe640 10 client.?.objecter ms_handle_connect 0x7f6f301457f0 2024-03-29T11:27:28.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.111+0000 7f6f177fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:28.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.112+0000 7f6f177fe640 1 -- 172.21.15.86:0/1236857684 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3193252477 0 0) 0x7f6f301387a0 con 0x7f6f300a9d70 2024-03-29T11:27:28.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.112+0000 7f6f177fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:28.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.112+0000 7f6f177fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:28.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.112+0000 7f6f177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:28.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.112+0000 7f6f177fe640 10 monclient(hunting): my global_id is 4441 2024-03-29T11:27:28.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.112+0000 7f6f177fe640 1 -- 172.21.15.86:0/1236857684 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f1c004eb0 con 0x7f6f300a9d70 2024-03-29T11:27:28.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.112+0000 7f6f177fe640 10 client.?.objecter ms_handle_connect 0x7f6f3013f650 2024-03-29T11:27:28.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.112+0000 7f6f177fe640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:28.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.112+0000 7f6f177fe640 1 -- 172.21.15.86:0/1236857684 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2384611040 0 0) 0x7f6f300ad7a0 con 0x7f6f301457f0 2024-03-29T11:27:28.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.112+0000 7f6f177fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:28.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.112+0000 7f6f177fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:28.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.112+0000 7f6f177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:28.110 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.112+0000 7f6f177fe640 10 monclient(hunting): my global_id is 4517 2024-03-29T11:27:28.110 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.112+0000 7f6f177fe640 1 -- 172.21.15.86:0/1236857684 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f301387a0 con 0x7f6f301457f0 2024-03-29T11:27:28.110 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.112+0000 7f6f177fe640 1 -- 172.21.15.86:0/1236857684 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 34526784 0 0) 0x7f6f301db340 con 0x7f6f3013f650 2024-03-29T11:27:28.110 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.112+0000 7f6f177fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:28.110 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.112+0000 7f6f177fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:28.110 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.112+0000 7f6f177fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:28.110 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.112+0000 7f6f177fe640 10 monclient(hunting): my global_id is 4449 2024-03-29T11:27:28.110 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.112+0000 7f6f177fe640 1 -- 172.21.15.86:0/1236857684 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6f300ad7a0 con 0x7f6f3013f650 2024-03-29T11:27:28.110 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.112+0000 7f6f177fe640 1 -- 172.21.15.86:0/1236857684 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4022628059 0 0) 0x7f6f301387a0 con 0x7f6f301457f0 2024-03-29T11:27:28.110 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.112+0000 7f6f177fe640 1 -- 172.21.15.86:0/1236857684 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f301db340 con 0x7f6f301457f0 2024-03-29T11:27:28.110 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.112+0000 7f6f177fe640 1 -- 172.21.15.86:0/1236857684 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f6f28001bb0 con 0x7f6f301457f0 2024-03-29T11:27:28.111 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.112+0000 7f6f177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:28.111 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.112+0000 7f6f177fe640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:28.111 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.112+0000 7f6f177fe640 10 monclient(hunting): dump: 2024-03-29T11:27:28.111 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:28.111 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:28.111 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:28.111 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:28.111 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:28.111 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:28.111 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:28.111 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:28.111 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:28.111 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:28.111 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.112+0000 7f6f177fe640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:28.112 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.112+0000 7f6f177fe640 1 -- 172.21.15.86:0/1236857684 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2010511287 0 0) 0x7f6f1c004eb0 con 0x7f6f300a9d70 2024-03-29T11:27:28.112 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.113+0000 7f6f177fe640 1 -- 172.21.15.86:0/1236857684 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f301387a0 con 0x7f6f300a9d70 2024-03-29T11:27:28.112 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.113+0000 7f6f177fe640 1 -- 172.21.15.86:0/1236857684 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f6f20001bb0 con 0x7f6f300a9d70 2024-03-29T11:27:28.112 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.113+0000 7f6f177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:28.112 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.113+0000 7f6f177fe640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:28.112 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.113+0000 7f6f177fe640 10 monclient(hunting): dump: 2024-03-29T11:27:28.112 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:28.112 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:28.112 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:28.112 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:28.112 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:28.112 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:28.112 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:28.112 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:28.112 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:28.113 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:28.113 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.113+0000 7f6f177fe640 1 -- 172.21.15.86:0/1236857684 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4063315486 0 0) 0x7f6f300ad7a0 con 0x7f6f3013f650 2024-03-29T11:27:28.113 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.113+0000 7f6f177fe640 1 -- 172.21.15.86:0/1236857684 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6f1c004eb0 con 0x7f6f3013f650 2024-03-29T11:27:28.113 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.113+0000 7f6f177fe640 1 -- 172.21.15.86:0/1236857684 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f6f24001750 con 0x7f6f3013f650 2024-03-29T11:27:28.113 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.113+0000 7f6f177fe640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:28.113 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.113+0000 7f6f177fe640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:28.113 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.113+0000 7f6f177fe640 10 monclient(hunting): dump: 2024-03-29T11:27:28.113 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:28.113 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:28.113 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:28.113 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:28.113 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:28.113 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:28.113 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:28.114 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:28.114 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:28.114 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:28.114 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.113+0000 7f6f177fe640 1 -- 172.21.15.86:0/1236857684 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2882805088 0 0) 0x7f6f301db340 con 0x7f6f301457f0 2024-03-29T11:27:28.114 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.113+0000 7f6f177fe640 1 -- 172.21.15.86:0/1236857684 >> v1:172.21.15.196:6789/0 conn(0x7f6f300a9d70 legacy=0x7f6f300ab600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:28.114 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.113+0000 7f6f177fe640 1 -- 172.21.15.86:0/1236857684 >> v1:172.21.15.86:6789/0 conn(0x7f6f3013f650 legacy=0x7f6f30137980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:28.114 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.113+0000 7f6f177fe640 10 monclient: _finish_hunting 0 2024-03-29T11:27:28.114 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.113+0000 7f6f177fe640 1 monclient: found mon.c 2024-03-29T11:27:28.114 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.113+0000 7f6f177fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:28.114 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.113+0000 7f6f177fe640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:27:28.114 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.113+0000 7f6f177fe640 1 -- 172.21.15.86:0/1236857684 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f301db7d0 con 0x7f6f301457f0 2024-03-29T11:27:28.115 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.113+0000 7f6f177fe640 10 monclient: _finish_auth 0 2024-03-29T11:27:28.115 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.113+0000 7f6f177fe640 10 monclient: _check_auth_tickets 2024-03-29T11:27:28.115 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.113+0000 7f6f177fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:28.115 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.113+0000 7f6f35732640 5 monclient: authenticate success, global_id 4517 2024-03-29T11:27:28.115 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.113+0000 7f6f35732640 10 monclient: _renew_subs 2024-03-29T11:27:28.115 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.113+0000 7f6f35732640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:27:28.115 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.113+0000 7f6f35732640 1 -- 172.21.15.86:0/1236857684 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f300abd10 con 0x7f6f301457f0 2024-03-29T11:27:28.115 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.113+0000 7f6f35732640 10 client.4517.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:28.115 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.113+0000 7f6f35732640 10 monclient: _renew_subs 2024-03-29T11:27:28.115 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.113+0000 7f6f35732640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:27:28.115 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.113+0000 7f6f35732640 1 -- 172.21.15.86:0/1236857684 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f30142d10 con 0x7f6f301457f0 2024-03-29T11:27:28.115 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.113+0000 7f6f35732640 1 librados: init done 2024-03-29T11:27:28.115 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.114+0000 7f6f35732640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:28.115 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.114+0000 7f6f35732640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:28.116 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.114+0000 7f6f35732640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:27:28.116 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.114+0000 7f6f35732640 1 -- 172.21.15.86:0/1236857684 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f301db9e0 con 0x7f6f301457f0 2024-03-29T11:27:28.116 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.115+0000 7f6f177fe640 1 -- 172.21.15.86:0/1236857684 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6f28002020 con 0x7f6f301457f0 2024-03-29T11:27:28.116 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.115+0000 7f6f177fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:28.116 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.115+0000 7f6f177fe640 1 -- 172.21.15.86:0/1236857684 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f6f28003970 con 0x7f6f301457f0 2024-03-29T11:27:28.116 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.115+0000 7f6f177fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:28.116 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.115+0000 7f6f177fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:28.116 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.115+0000 7f6f177fe640 10 monclient: dump: 2024-03-29T11:27:28.116 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:28.117 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:28.117 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:28.117 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:28.117 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:28.117 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:28.117 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:28.117 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:28.117 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:28.117 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:28.117 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.115+0000 7f6f2cff9640 4 set_mon_vals no callback set 2024-03-29T11:27:28.117 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.116+0000 7f6f177fe640 1 -- 172.21.15.86:0/1236857684 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f6f28003b90 con 0x7f6f301457f0 2024-03-29T11:27:28.117 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.117+0000 7f6f177fe640 1 -- 172.21.15.86:0/1236857684 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f6f28083800 con 0x7f6f301457f0 2024-03-29T11:27:28.117 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.117+0000 7f6f177fe640 10 client.4517.objecter ms_dispatch 0x7f6f30000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:28.117 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.117+0000 7f6f177fe640 3 client.4517.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:28.117 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.117+0000 7f6f177fe640 3 client.4517.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:28.118 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.117+0000 7f6f177fe640 20 client.4517.objecter dump_active .. 0 homeless 2024-03-29T11:27:28.118 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.117+0000 7f6f2effd640 10 monclient: get_auth_request con 0x7f6f1c06e0f0 auth_method 0 2024-03-29T11:27:28.118 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.117+0000 7f6f177fe640 10 client.4517.objecter ms_handle_connect 0x7f6f1c06e0f0 2024-03-29T11:27:28.118 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.122+0000 7f6f177fe640 1 -- 172.21.15.86:0/1236857684 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6f2804f140 con 0x7f6f301457f0 2024-03-29T11:27:28.118 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.122+0000 7f6f177fe640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:28.118 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.122+0000 7f6f177fe640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:28.402 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.413+0000 7f6f35732640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:28.402 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.414+0000 7f6f35732640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:28.402 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.414+0000 7f6f35732640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:27:28.402 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.414+0000 7f6f35732640 1 -- 172.21.15.86:0/1236857684 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f301db4f0 con 0x7f6f301457f0 2024-03-29T11:27:28.402 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.414+0000 7f6f177fe640 1 -- 172.21.15.86:0/1236857684 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f6f301db4f0 con 0x7f6f301457f0 2024-03-29T11:27:28.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.414+0000 7f6f177fe640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:28.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.414+0000 7f6f177fe640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:28.403 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:27:28.403 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:28.406 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.417+0000 7f6f35732640 10 librados: watch_flush enter 2024-03-29T11:27:28.406 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.417+0000 7f6f35732640 10 librados: watch_flush exit 2024-03-29T11:27:28.406 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.417+0000 7f6f35732640 20 client.4517.objecter shutdown clearing up homeless session... 2024-03-29T11:27:28.406 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.417+0000 7f6f35732640 10 client.4517.objecter successfully canceled tick 2024-03-29T11:27:28.406 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.417+0000 7f6f35732640 1 -- 172.21.15.86:0/1236857684 >> v1:172.21.15.196:6816/588110318 conn(0x7f6f1c06e0f0 legacy=0x7f6f1c0705b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:28.406 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.417+0000 7f6f35732640 10 monclient: shutdown 2024-03-29T11:27:28.406 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.417+0000 7f6f35732640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:28.406 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.417+0000 7f6f35732640 1 -- 172.21.15.86:0/1236857684 >> v1:172.21.15.86:6790/0 conn(0x7f6f301457f0 legacy=0x7f6f30138090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:28.406 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.417+0000 7f6f35732640 1 -- 172.21.15.86:0/1236857684 shutdown_connections 2024-03-29T11:27:28.407 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.417+0000 7f6f35732640 1 -- 172.21.15.86:0/1236857684 >> 172.21.15.86:0/1236857684 conn(0x7f6f30066a00 msgr2=0x7f6f300b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:28.407 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.417+0000 7f6f35732640 1 -- 172.21.15.86:0/1236857684 shutdown_connections 2024-03-29T11:27:28.407 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.417+0000 7f6f35732640 1 -- 172.21.15.86:0/1236857684 wait complete. 2024-03-29T11:27:28.407 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:28.418+0000 7f6f35732640 1 librados: shutdown 2024-03-29T11:27:28.419 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:35.227 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:35.378 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.390+0000 7f3f56f72640 10 monclient: get_monmap_and_config 2024-03-29T11:27:35.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.390+0000 7f3f56f72640 10 monclient: build_initial_monmap 2024-03-29T11:27:35.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.390+0000 7f3f56f72640 10 monclient: monmap: 2024-03-29T11:27:35.379 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:27:35.379 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:35.379 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:27:35.391382+0000 2024-03-29T11:27:35.379 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:27:35.391382+0000 2024-03-29T11:27:35.379 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:35.379 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:35.379 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:27:35.379 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:27:35.379 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:27:35.379 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:35.380 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.391+0000 7f3f56f72640 1 Processor -- start 2024-03-29T11:27:35.380 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.391+0000 7f3f56f72640 1 -- start start 2024-03-29T11:27:35.380 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.391+0000 7f3f56f72640 10 monclient: init 2024-03-29T11:27:35.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.392+0000 7f3f56f72640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:35.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.392+0000 7f3f56f72640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:35.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.392+0000 7f3f56f72640 10 monclient(hunting): picked mon.noname-a con 0x7f3f500b1f60 addr v1:172.21.15.86:6789/0 2024-03-29T11:27:35.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.392+0000 7f3f56f72640 10 monclient(hunting): picked mon.noname-c con 0x7f3f50071eb0 addr v1:172.21.15.196:6789/0 2024-03-29T11:27:35.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.392+0000 7f3f56f72640 10 monclient(hunting): picked mon.noname-b con 0x7f3f50143d10 addr v1:172.21.15.86:6790/0 2024-03-29T11:27:35.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.392+0000 7f3f56f72640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f5006ef80 con 0x7f3f500b1f60 2024-03-29T11:27:35.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.392+0000 7f3f56f72640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f5006f260 con 0x7f3f50071eb0 2024-03-29T11:27:35.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.392+0000 7f3f56f72640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f500b40c0 con 0x7f3f50143d10 2024-03-29T11:27:35.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.392+0000 7f3f56f72640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:35.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.392+0000 7f3f56f72640 10 monclient(hunting): authenticate will time out at 1279.710440s 2024-03-29T11:27:35.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.398+0000 7f3f4ffff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f3f50071eb0 0x7f3f50146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:47631/0 (socket says 172.21.15.86:47631) 2024-03-29T11:27:35.387 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.398+0000 7f3f4ffff640 1 -- 172.21.15.86:0/1746772221 learned_addr learned my addr 172.21.15.86:0/1746772221 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:27:35.389 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.399+0000 7f3f4f7fe640 1 -- 172.21.15.86:0/1746772221 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1805527508 0 0) 0x7f3f500b40c0 con 0x7f3f50143d10 2024-03-29T11:27:35.389 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.399+0000 7f3f4f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:27:35.389 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.399+0000 7f3f4f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:35.389 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.399+0000 7f3f4f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:35.389 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.399+0000 7f3f4f7fe640 10 monclient(hunting): my global_id is 4520 2024-03-29T11:27:35.389 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.399+0000 7f3f4f7fe640 1 -- 172.21.15.86:0/1746772221 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f38003520 con 0x7f3f50143d10 2024-03-29T11:27:35.389 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.400+0000 7f3f4f7fe640 1 -- 172.21.15.86:0/1746772221 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 886203250 0 0) 0x7f3f38003520 con 0x7f3f50143d10 2024-03-29T11:27:35.389 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.400+0000 7f3f4f7fe640 1 -- 172.21.15.86:0/1746772221 >> v1:172.21.15.196:6789/0 conn(0x7f3f50071eb0 legacy=0x7f3f50146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:35.389 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.400+0000 7f3f4f7fe640 1 -- 172.21.15.86:0/1746772221 >> v1:172.21.15.86:6789/0 conn(0x7f3f500b1f60 legacy=0x7f3f50143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:35.389 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.400+0000 7f3f4f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:27:35.389 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.400+0000 7f3f4f7fe640 1 monclient: found mon.noname-b 2024-03-29T11:27:35.389 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.400+0000 7f3f4f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:35.390 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.400+0000 7f3f4f7fe640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:27:35.390 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.400+0000 7f3f4f7fe640 1 -- 172.21.15.86:0/1746772221 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f50155d70 con 0x7f3f50143d10 2024-03-29T11:27:35.390 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.400+0000 7f3f4f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:27:35.390 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.400+0000 7f3f4f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:27:35.390 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.400+0000 7f3f4f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:35.390 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.400+0000 7f3f4f7fe640 1 -- 172.21.15.86:0/1746772221 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f3f40001690 con 0x7f3f50143d10 2024-03-29T11:27:35.390 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.400+0000 7f3f4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:35.390 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.400+0000 7f3f4f7fe640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:35.390 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.400+0000 7f3f4f7fe640 10 monclient: dump: 2024-03-29T11:27:35.390 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:35.390 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:35.390 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:35.390 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:35.390 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:35.390 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:35.391 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:35.391 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:35.391 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:35.391 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:35.391 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.400+0000 7f3f4f7fe640 1 -- 172.21.15.86:0/1746772221 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f40001920 con 0x7f3f50143d10 2024-03-29T11:27:35.391 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.400+0000 7f3f4f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:35.391 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.400+0000 7f3f4f7fe640 1 -- 172.21.15.86:0/1746772221 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f3f40001cb0 con 0x7f3f50143d10 2024-03-29T11:27:35.391 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.400+0000 7f3f4f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:35.391 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.400+0000 7f3f4f7fe640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:35.391 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.400+0000 7f3f4f7fe640 10 monclient: dump: 2024-03-29T11:27:35.391 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:35.391 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:35.391 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:35.391 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:35.391 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:35.392 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:35.392 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:35.392 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:35.392 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:35.392 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:35.392 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.400+0000 7f3f56f72640 5 monclient: authenticate success, global_id 4520 2024-03-29T11:27:35.392 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.400+0000 7f3f56f72640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:35.392 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.400+0000 7f3f56f72640 4 set_mon_vals no callback set 2024-03-29T11:27:35.393 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.401+0000 7f3f56f72640 10 monclient: shutdown 2024-03-29T11:27:35.394 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.401+0000 7f3f56f72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:35.394 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.401+0000 7f3f56f72640 1 -- 172.21.15.86:0/1746772221 >> v1:172.21.15.86:6790/0 conn(0x7f3f50143d10 legacy=0x7f3f50146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:35.394 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.401+0000 7f3f56f72640 1 -- 172.21.15.86:0/1746772221 shutdown_connections 2024-03-29T11:27:35.394 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.401+0000 7f3f56f72640 1 -- 172.21.15.86:0/1746772221 >> 172.21.15.86:0/1746772221 conn(0x7f3f50066a00 msgr2=0x7f3f50066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:35.394 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.402+0000 7f3f56f72640 1 -- 172.21.15.86:0/1746772221 shutdown_connections 2024-03-29T11:27:35.394 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.402+0000 7f3f56f72640 1 -- 172.21.15.86:0/1746772221 wait complete. 2024-03-29T11:27:35.394 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.403+0000 7f3f56f72640 10 monclient: build_initial_monmap 2024-03-29T11:27:35.394 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.403+0000 7f3f56f72640 10 monclient: monmap: 2024-03-29T11:27:35.394 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:27:35.394 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:35.394 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:27:35.394 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:27:35.394 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:35.395 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:35.395 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:27:35.395 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:27:35.395 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:27:35.395 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:35.395 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.403+0000 7f3f56f72640 1 librados: starting msgr at 2024-03-29T11:27:35.395 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.403+0000 7f3f56f72640 1 librados: starting objecter 2024-03-29T11:27:35.395 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.403+0000 7f3f56f72640 1 Processor -- start 2024-03-29T11:27:35.397 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.403+0000 7f3f56f72640 1 -- start start 2024-03-29T11:27:35.397 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.403+0000 7f3f56f72640 1 librados: setting wanted keys 2024-03-29T11:27:35.397 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.403+0000 7f3f56f72640 1 librados: calling monclient init 2024-03-29T11:27:35.397 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.403+0000 7f3f56f72640 10 monclient: init 2024-03-29T11:27:35.397 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.404+0000 7f3f56f72640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:35.397 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.404+0000 7f3f56f72640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:27:35.397 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.404+0000 7f3f56f72640 10 monclient(hunting): picked mon.noname-b con 0x7f3f500b1f60 addr v1:172.21.15.196:6789/0 2024-03-29T11:27:35.398 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.404+0000 7f3f56f72640 10 monclient(hunting): picked mon.noname-a con 0x7f3f50143d10 addr v1:172.21.15.86:6789/0 2024-03-29T11:27:35.398 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.404+0000 7f3f56f72640 10 monclient(hunting): picked mon.noname-c con 0x7f3f500b17d0 addr v1:172.21.15.86:6790/0 2024-03-29T11:27:35.398 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.404+0000 7f3f56f72640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f50142e60 con 0x7f3f50143d10 2024-03-29T11:27:35.398 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.404+0000 7f3f56f72640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f50070150 con 0x7f3f500b1f60 2024-03-29T11:27:35.398 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.404+0000 7f3f56f72640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f50070430 con 0x7f3f500b17d0 2024-03-29T11:27:35.398 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.404+0000 7f3f56f72640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:35.398 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.404+0000 7f3f56f72640 10 monclient(hunting): authenticate will time out at 1279.722868s 2024-03-29T11:27:35.398 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.408+0000 7f3f54d0e640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f3f500b1f60 0x7f3f50134030 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:47693/0 (socket says 172.21.15.86:47693) 2024-03-29T11:27:35.398 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.408+0000 7f3f54d0e640 1 -- 172.21.15.86:0/583179849 learned_addr learned my addr 172.21.15.86:0/583179849 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:27:35.402 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.414+0000 7f3f4d7fa640 10 client.?.objecter ms_handle_connect 0x7f3f500b1f60 2024-03-29T11:27:35.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.414+0000 7f3f4d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:35.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.414+0000 7f3f4d7fa640 10 client.?.objecter ms_handle_connect 0x7f3f50143d10 2024-03-29T11:27:35.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.414+0000 7f3f4d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:35.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.414+0000 7f3f4d7fa640 1 -- 172.21.15.86:0/583179849 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 948244207 0 0) 0x7f3f50070150 con 0x7f3f500b1f60 2024-03-29T11:27:35.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.414+0000 7f3f4d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:27:35.403 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.414+0000 7f3f4d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:35.404 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.414+0000 7f3f4d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:35.404 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.414+0000 7f3f4d7fa640 10 monclient(hunting): my global_id is 4444 2024-03-29T11:27:35.404 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.414+0000 7f3f4d7fa640 1 -- 172.21.15.86:0/583179849 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f3c004eb0 con 0x7f3f500b1f60 2024-03-29T11:27:35.404 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.414+0000 7f3f4d7fa640 1 -- 172.21.15.86:0/583179849 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1225739543 0 0) 0x7f3f50142e60 con 0x7f3f50143d10 2024-03-29T11:27:35.404 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.414+0000 7f3f4d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:27:35.404 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.414+0000 7f3f4d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:35.404 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.414+0000 7f3f4d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:35.404 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.414+0000 7f3f4d7fa640 10 monclient(hunting): my global_id is 4452 2024-03-29T11:27:35.404 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.414+0000 7f3f4d7fa640 1 -- 172.21.15.86:0/583179849 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f50070150 con 0x7f3f50143d10 2024-03-29T11:27:35.404 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.415+0000 7f3f4d7fa640 1 -- 172.21.15.86:0/583179849 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1294947529 0 0) 0x7f3f50070150 con 0x7f3f50143d10 2024-03-29T11:27:35.409 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.418+0000 7f3f4d7fa640 1 -- 172.21.15.86:0/583179849 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f50142e60 con 0x7f3f50143d10 2024-03-29T11:27:35.409 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.418+0000 7f3f4d7fa640 1 -- 172.21.15.86:0/583179849 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1031824358 0 0) 0x7f3f3c004eb0 con 0x7f3f500b1f60 2024-03-29T11:27:35.409 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.418+0000 7f3f4d7fa640 1 -- 172.21.15.86:0/583179849 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3f50070150 con 0x7f3f500b1f60 2024-03-29T11:27:35.409 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.418+0000 7f3f4d7fa640 10 client.?.objecter ms_handle_connect 0x7f3f500b17d0 2024-03-29T11:27:35.409 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.418+0000 7f3f4d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:35.409 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.418+0000 7f3f4d7fa640 1 -- 172.21.15.86:0/583179849 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f3f48001810 con 0x7f3f50143d10 2024-03-29T11:27:35.409 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.418+0000 7f3f4d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:35.409 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.419+0000 7f3f4d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:35.409 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.419+0000 7f3f4d7fa640 10 monclient(hunting): dump: 2024-03-29T11:27:35.409 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:35.409 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:35.409 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:35.409 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:35.409 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:35.410 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:35.410 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:35.410 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:35.410 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:35.410 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:35.410 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.419+0000 7f3f4d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:35.410 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.419+0000 7f3f4d7fa640 1 -- 172.21.15.86:0/583179849 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f3f440018e0 con 0x7f3f500b1f60 2024-03-29T11:27:35.410 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.419+0000 7f3f4d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:35.410 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.419+0000 7f3f4d7fa640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:35.410 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.419+0000 7f3f4d7fa640 10 monclient(hunting): dump: 2024-03-29T11:27:35.410 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:35.410 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:35.410 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:35.411 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:35.411 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:35.411 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:35.411 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:35.411 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:35.411 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:35.411 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:35.411 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.419+0000 7f3f4d7fa640 1 -- 172.21.15.86:0/583179849 <== mon.0 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1376327049 0 0) 0x7f3f50142e60 con 0x7f3f50143d10 2024-03-29T11:27:35.411 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.419+0000 7f3f4d7fa640 1 -- 172.21.15.86:0/583179849 >> v1:172.21.15.86:6790/0 conn(0x7f3f500b17d0 legacy=0x7f3f501eed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:35.411 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.419+0000 7f3f4d7fa640 1 -- 172.21.15.86:0/583179849 >> v1:172.21.15.196:6789/0 conn(0x7f3f500b1f60 legacy=0x7f3f50134030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:35.411 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.419+0000 7f3f4d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:27:35.411 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.419+0000 7f3f4d7fa640 1 monclient: found mon.a 2024-03-29T11:27:35.411 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.419+0000 7f3f4d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:35.411 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.419+0000 7f3f4d7fa640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:27:35.411 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.419+0000 7f3f4d7fa640 1 -- 172.21.15.86:0/583179849 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f50070710 con 0x7f3f50143d10 2024-03-29T11:27:35.412 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.419+0000 7f3f4d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:27:35.412 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.420+0000 7f3f4d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:27:35.412 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.420+0000 7f3f4d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:35.412 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.420+0000 7f3f56f72640 5 monclient: authenticate success, global_id 4452 2024-03-29T11:27:35.412 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.420+0000 7f3f56f72640 10 monclient: _renew_subs 2024-03-29T11:27:35.412 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.420+0000 7f3f56f72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:27:35.412 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.420+0000 7f3f56f72640 1 -- 172.21.15.86:0/583179849 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f50155370 con 0x7f3f50143d10 2024-03-29T11:27:35.412 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.420+0000 7f3f56f72640 10 client.4452.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:35.412 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.420+0000 7f3f56f72640 10 monclient: _renew_subs 2024-03-29T11:27:35.412 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.420+0000 7f3f56f72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:27:35.412 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.420+0000 7f3f56f72640 1 -- 172.21.15.86:0/583179849 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f50070ef0 con 0x7f3f50143d10 2024-03-29T11:27:35.412 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.420+0000 7f3f56f72640 1 librados: init done 2024-03-29T11:27:35.412 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.420+0000 7f3f4d7fa640 1 -- 172.21.15.86:0/583179849 <== mon.0 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3f48001d80 con 0x7f3f50143d10 2024-03-29T11:27:35.412 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.420+0000 7f3f4d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:35.412 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.420+0000 7f3f4d7fa640 1 -- 172.21.15.86:0/583179849 <== mon.0 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f3f48004e10 con 0x7f3f50143d10 2024-03-29T11:27:35.413 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.420+0000 7f3f4d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:35.413 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.420+0000 7f3f4d7fa640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:35.413 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.420+0000 7f3f4d7fa640 10 monclient: dump: 2024-03-29T11:27:35.413 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:35.413 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:35.413 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:35.413 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:35.413 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:35.413 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:35.413 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:35.413 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:35.413 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:35.413 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:35.414 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.421+0000 7f3f56f72640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:35.414 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.421+0000 7f3f4effd640 4 set_mon_vals no callback set 2024-03-29T11:27:35.415 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.422+0000 7f3f4d7fa640 1 -- 172.21.15.86:0/583179849 <== mon.0 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f3f4801ade0 con 0x7f3f50143d10 2024-03-29T11:27:35.415 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.423+0000 7f3f4d7fa640 1 -- 172.21.15.86:0/583179849 <== mon.0 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f3f48084b40 con 0x7f3f50143d10 2024-03-29T11:27:35.415 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.423+0000 7f3f4d7fa640 10 client.4452.objecter ms_dispatch 0x7f3f50000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:35.415 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.423+0000 7f3f4d7fa640 3 client.4452.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:35.415 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.423+0000 7f3f4d7fa640 3 client.4452.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:35.415 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.423+0000 7f3f4d7fa640 20 client.4452.objecter dump_active .. 0 homeless 2024-03-29T11:27:35.415 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.425+0000 7f3f56f72640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:35.415 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.425+0000 7f3f56f72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:27:35.415 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.425+0000 7f3f56f72640 1 -- 172.21.15.86:0/583179849 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f50070c30 con 0x7f3f50143d10 2024-03-29T11:27:35.415 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.426+0000 7f3f54d0e640 10 monclient: get_auth_request con 0x7f3f3c06e060 auth_method 0 2024-03-29T11:27:35.420 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.432+0000 7f3f4d7fa640 10 client.4452.objecter ms_handle_connect 0x7f3f3c06e060 2024-03-29T11:27:35.423 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.434+0000 7f3f4d7fa640 1 -- 172.21.15.86:0/583179849 <== mon.0 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3f50070c30 con 0x7f3f50143d10 2024-03-29T11:27:35.423 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.434+0000 7f3f4d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:35.423 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.434+0000 7f3f4d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:35.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.757+0000 7f3f56f72640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:35.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.757+0000 7f3f56f72640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:35.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.757+0000 7f3f56f72640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:27:35.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.757+0000 7f3f56f72640 1 -- 172.21.15.86:0/583179849 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f50070c30 con 0x7f3f50143d10 2024-03-29T11:27:35.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.758+0000 7f3f4d7fa640 1 -- 172.21.15.86:0/583179849 <== mon.0 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f3f50070c30 con 0x7f3f50143d10 2024-03-29T11:27:35.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.758+0000 7f3f4d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:35.747 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.758+0000 7f3f4d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:35.747 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:27:35.747 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:35.749 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.760+0000 7f3f56f72640 10 librados: watch_flush enter 2024-03-29T11:27:35.750 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.760+0000 7f3f56f72640 10 librados: watch_flush exit 2024-03-29T11:27:35.750 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.760+0000 7f3f56f72640 20 client.4452.objecter shutdown clearing up homeless session... 2024-03-29T11:27:35.750 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.760+0000 7f3f56f72640 10 client.4452.objecter successfully canceled tick 2024-03-29T11:27:35.750 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.760+0000 7f3f56f72640 1 -- 172.21.15.86:0/583179849 >> v1:172.21.15.196:6816/588110318 conn(0x7f3f3c06e060 legacy=0x7f3f3c070520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:35.750 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.760+0000 7f3f56f72640 10 monclient: shutdown 2024-03-29T11:27:35.750 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.760+0000 7f3f56f72640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:35.750 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.760+0000 7f3f56f72640 1 -- 172.21.15.86:0/583179849 >> v1:172.21.15.86:6789/0 conn(0x7f3f50143d10 legacy=0x7f3f50142750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:35.750 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.761+0000 7f3f56f72640 1 -- 172.21.15.86:0/583179849 shutdown_connections 2024-03-29T11:27:35.750 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.761+0000 7f3f56f72640 1 -- 172.21.15.86:0/583179849 >> 172.21.15.86:0/583179849 conn(0x7f3f50066a00 msgr2=0x7f3f500b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:35.750 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.761+0000 7f3f56f72640 1 -- 172.21.15.86:0/583179849 shutdown_connections 2024-03-29T11:27:35.751 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.761+0000 7f3f56f72640 1 -- 172.21.15.86:0/583179849 wait complete. 2024-03-29T11:27:35.751 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:35.761+0000 7f3f56f72640 1 librados: shutdown 2024-03-29T11:27:35.763 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:42.570 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:42.728 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.739+0000 7fba2e7e2640 10 monclient: get_monmap_and_config 2024-03-29T11:27:42.728 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.739+0000 7fba2e7e2640 10 monclient: build_initial_monmap 2024-03-29T11:27:42.728 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.739+0000 7fba2e7e2640 10 monclient: monmap: 2024-03-29T11:27:42.728 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:27:42.728 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:42.728 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:27:42.740427+0000 2024-03-29T11:27:42.728 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:27:42.740427+0000 2024-03-29T11:27:42.728 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:42.728 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:42.728 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:27:42.728 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:27:42.729 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:27:42.729 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:42.731 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.741+0000 7fba2e7e2640 1 Processor -- start 2024-03-29T11:27:42.731 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.741+0000 7fba2e7e2640 1 -- start start 2024-03-29T11:27:42.731 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.741+0000 7fba2e7e2640 10 monclient: init 2024-03-29T11:27:42.731 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.741+0000 7fba2e7e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:42.731 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.741+0000 7fba2e7e2640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:42.731 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.741+0000 7fba2e7e2640 10 monclient(hunting): picked mon.noname-a con 0x7fba280b1f60 addr v1:172.21.15.86:6789/0 2024-03-29T11:27:42.731 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.742+0000 7fba2e7e2640 10 monclient(hunting): picked mon.noname-c con 0x7fba28071eb0 addr v1:172.21.15.196:6789/0 2024-03-29T11:27:42.731 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.742+0000 7fba2e7e2640 10 monclient(hunting): picked mon.noname-b con 0x7fba28143d10 addr v1:172.21.15.86:6790/0 2024-03-29T11:27:42.732 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.742+0000 7fba2e7e2640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba2806ef80 con 0x7fba280b1f60 2024-03-29T11:27:42.732 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.742+0000 7fba2e7e2640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba2806f260 con 0x7fba28071eb0 2024-03-29T11:27:42.732 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.742+0000 7fba2e7e2640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba280b40c0 con 0x7fba28143d10 2024-03-29T11:27:42.732 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.742+0000 7fba2e7e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:42.732 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.742+0000 7fba2e7e2640 10 monclient(hunting): authenticate will time out at 1287.060056s 2024-03-29T11:27:42.732 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.743+0000 7fba2cd7f640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7fba28143d10 0x7fba28146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:60934/0 (socket says 172.21.15.86:60934) 2024-03-29T11:27:42.732 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.743+0000 7fba2cd7f640 1 -- 172.21.15.86:0/2505638306 learned_addr learned my addr 172.21.15.86:0/2505638306 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:27:42.736 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.747+0000 7fba26ffd640 1 -- 172.21.15.86:0/2505638306 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 904152270 0 0) 0x7fba280b40c0 con 0x7fba28143d10 2024-03-29T11:27:42.736 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.747+0000 7fba26ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:42.736 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.747+0000 7fba26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:42.736 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.747+0000 7fba26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:42.736 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.747+0000 7fba26ffd640 10 monclient(hunting): my global_id is 4526 2024-03-29T11:27:42.736 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.747+0000 7fba26ffd640 1 -- 172.21.15.86:0/2505638306 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba10003520 con 0x7fba28143d10 2024-03-29T11:27:42.739 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.747+0000 7fba26ffd640 1 -- 172.21.15.86:0/2505638306 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3600466484 0 0) 0x7fba2806f260 con 0x7fba28071eb0 2024-03-29T11:27:42.739 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.747+0000 7fba26ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:27:42.739 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.747+0000 7fba26ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:42.739 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.747+0000 7fba26ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:42.740 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.747+0000 7fba26ffd640 10 monclient(hunting): my global_id is 4447 2024-03-29T11:27:42.740 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.747+0000 7fba26ffd640 1 -- 172.21.15.86:0/2505638306 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba280b40c0 con 0x7fba28071eb0 2024-03-29T11:27:42.740 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.747+0000 7fba26ffd640 1 -- 172.21.15.86:0/2505638306 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1577604384 0 0) 0x7fba10003520 con 0x7fba28143d10 2024-03-29T11:27:42.740 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.747+0000 7fba26ffd640 1 -- 172.21.15.86:0/2505638306 >> v1:172.21.15.196:6789/0 conn(0x7fba28071eb0 legacy=0x7fba28146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:42.740 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.747+0000 7fba26ffd640 1 -- 172.21.15.86:0/2505638306 >> v1:172.21.15.86:6789/0 conn(0x7fba280b1f60 legacy=0x7fba28143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:42.740 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.747+0000 7fba26ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:27:42.740 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.747+0000 7fba26ffd640 1 monclient: found mon.noname-b 2024-03-29T11:27:42.740 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.747+0000 7fba26ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:42.740 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.747+0000 7fba26ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:27:42.740 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.747+0000 7fba26ffd640 1 -- 172.21.15.86:0/2505638306 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba28155d70 con 0x7fba28143d10 2024-03-29T11:27:42.740 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.747+0000 7fba26ffd640 10 monclient: _finish_auth 0 2024-03-29T11:27:42.740 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.747+0000 7fba26ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:27:42.740 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.747+0000 7fba26ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:42.740 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.748+0000 7fba26ffd640 1 -- 172.21.15.86:0/2505638306 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fba20001610 con 0x7fba28143d10 2024-03-29T11:27:42.740 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.748+0000 7fba26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:42.741 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.748+0000 7fba26ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:42.741 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.748+0000 7fba26ffd640 10 monclient: dump: 2024-03-29T11:27:42.741 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:42.741 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:42.741 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:42.741 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:42.741 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:42.741 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:42.741 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:42.741 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:42.741 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:42.741 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:42.741 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.748+0000 7fba26ffd640 1 -- 172.21.15.86:0/2505638306 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba20001830 con 0x7fba28143d10 2024-03-29T11:27:42.741 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.748+0000 7fba26ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:42.741 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.748+0000 7fba26ffd640 1 -- 172.21.15.86:0/2505638306 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fba20001bc0 con 0x7fba28143d10 2024-03-29T11:27:42.742 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.748+0000 7fba26ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:42.742 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.748+0000 7fba26ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:42.742 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.748+0000 7fba26ffd640 10 monclient: dump: 2024-03-29T11:27:42.742 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:42.742 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:42.742 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:42.742 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:42.742 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:42.742 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:42.742 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:42.742 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:42.742 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:42.742 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:42.742 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.748+0000 7fba2e7e2640 5 monclient: authenticate success, global_id 4526 2024-03-29T11:27:42.742 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.748+0000 7fba2e7e2640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:42.743 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.748+0000 7fba2e7e2640 4 set_mon_vals no callback set 2024-03-29T11:27:42.743 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.749+0000 7fba2e7e2640 10 monclient: shutdown 2024-03-29T11:27:42.743 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.749+0000 7fba2e7e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:42.743 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.749+0000 7fba2e7e2640 1 -- 172.21.15.86:0/2505638306 >> v1:172.21.15.86:6790/0 conn(0x7fba28143d10 legacy=0x7fba28146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:42.743 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.749+0000 7fba2e7e2640 1 -- 172.21.15.86:0/2505638306 shutdown_connections 2024-03-29T11:27:42.743 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.749+0000 7fba2e7e2640 1 -- 172.21.15.86:0/2505638306 >> 172.21.15.86:0/2505638306 conn(0x7fba28066a00 msgr2=0x7fba28066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:42.743 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.750+0000 7fba2e7e2640 1 -- 172.21.15.86:0/2505638306 shutdown_connections 2024-03-29T11:27:42.743 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.750+0000 7fba2e7e2640 1 -- 172.21.15.86:0/2505638306 wait complete. 2024-03-29T11:27:42.744 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.751+0000 7fba2e7e2640 10 monclient: build_initial_monmap 2024-03-29T11:27:42.744 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.751+0000 7fba2e7e2640 10 monclient: monmap: 2024-03-29T11:27:42.744 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:27:42.744 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:42.745 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:27:42.745 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:27:42.745 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:42.745 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:42.745 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:27:42.745 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:27:42.745 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:27:42.745 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:42.745 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.752+0000 7fba2e7e2640 1 librados: starting msgr at 2024-03-29T11:27:42.745 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.752+0000 7fba2e7e2640 1 librados: starting objecter 2024-03-29T11:27:42.745 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.752+0000 7fba2e7e2640 1 Processor -- start 2024-03-29T11:27:42.745 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.752+0000 7fba2e7e2640 1 -- start start 2024-03-29T11:27:42.745 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.752+0000 7fba2e7e2640 1 librados: setting wanted keys 2024-03-29T11:27:42.745 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.752+0000 7fba2e7e2640 1 librados: calling monclient init 2024-03-29T11:27:42.745 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.752+0000 7fba2e7e2640 10 monclient: init 2024-03-29T11:27:42.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.753+0000 7fba2e7e2640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:42.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.753+0000 7fba2e7e2640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:27:42.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.753+0000 7fba2e7e2640 10 monclient(hunting): picked mon.noname-a con 0x7fba280b1f60 addr v1:172.21.15.86:6789/0 2024-03-29T11:27:42.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.753+0000 7fba2e7e2640 10 monclient(hunting): picked mon.noname-c con 0x7fba28143d10 addr v1:172.21.15.86:6790/0 2024-03-29T11:27:42.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.753+0000 7fba2e7e2640 10 monclient(hunting): picked mon.noname-b con 0x7fba280aa530 addr v1:172.21.15.196:6789/0 2024-03-29T11:27:42.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.753+0000 7fba2e7e2640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba280ac840 con 0x7fba280b1f60 2024-03-29T11:27:42.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.753+0000 7fba2e7e2640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba280acab0 con 0x7fba280aa530 2024-03-29T11:27:42.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.753+0000 7fba2e7e2640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fba280ab200 con 0x7fba28143d10 2024-03-29T11:27:42.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.753+0000 7fba2e7e2640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:42.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.753+0000 7fba2e7e2640 10 monclient(hunting): authenticate will time out at 1287.071569s 2024-03-29T11:27:42.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.753+0000 7fba277fe640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7fba28143d10 0x7fba281e7230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:42222/0 (socket says 172.21.15.86:42222) 2024-03-29T11:27:42.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.753+0000 7fba277fe640 1 -- 172.21.15.86:0/3096850506 learned_addr learned my addr 172.21.15.86:0/3096850506 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:27:42.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.753+0000 7fba24ff9640 10 client.?.objecter ms_handle_connect 0x7fba28143d10 2024-03-29T11:27:42.746 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.753+0000 7fba24ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:42.747 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.756+0000 7fba24ff9640 1 -- 172.21.15.86:0/3096850506 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1605481189 0 0) 0x7fba280ab200 con 0x7fba28143d10 2024-03-29T11:27:42.747 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.756+0000 7fba24ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:27:42.747 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.756+0000 7fba24ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:42.747 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.756+0000 7fba24ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:42.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.756+0000 7fba24ff9640 10 monclient(hunting): my global_id is 4529 2024-03-29T11:27:42.748 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.756+0000 7fba24ff9640 1 -- 172.21.15.86:0/3096850506 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba14004eb0 con 0x7fba28143d10 2024-03-29T11:27:42.751 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.757+0000 7fba24ff9640 10 client.?.objecter ms_handle_connect 0x7fba280b1f60 2024-03-29T11:27:42.752 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.757+0000 7fba24ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:42.752 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.757+0000 7fba24ff9640 1 -- 172.21.15.86:0/3096850506 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3481433018 0 0) 0x7fba14004eb0 con 0x7fba28143d10 2024-03-29T11:27:42.752 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.757+0000 7fba24ff9640 1 -- 172.21.15.86:0/3096850506 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba14005300 con 0x7fba28143d10 2024-03-29T11:27:42.752 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.757+0000 7fba24ff9640 1 -- 172.21.15.86:0/3096850506 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2692172683 0 0) 0x7fba280ac840 con 0x7fba280b1f60 2024-03-29T11:27:42.752 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.757+0000 7fba24ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:27:42.752 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.757+0000 7fba24ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:42.752 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.757+0000 7fba24ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:42.752 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.757+0000 7fba24ff9640 10 monclient(hunting): my global_id is 4458 2024-03-29T11:27:42.752 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.757+0000 7fba24ff9640 1 -- 172.21.15.86:0/3096850506 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fba14004eb0 con 0x7fba280b1f60 2024-03-29T11:27:42.752 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.757+0000 7fba24ff9640 1 -- 172.21.15.86:0/3096850506 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fba1c0019f0 con 0x7fba28143d10 2024-03-29T11:27:42.752 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.757+0000 7fba24ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:42.752 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.757+0000 7fba24ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:42.752 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.757+0000 7fba24ff9640 10 monclient(hunting): dump: 2024-03-29T11:27:42.753 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:42.753 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:42.753 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:42.753 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:42.753 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:42.753 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:42.753 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:42.753 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:42.753 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:42.753 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:42.753 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.757+0000 7fba24ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:42.753 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.757+0000 7fba24ff9640 1 -- 172.21.15.86:0/3096850506 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 209343266 0 0) 0x7fba14004eb0 con 0x7fba280b1f60 2024-03-29T11:27:42.753 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.757+0000 7fba24ff9640 1 -- 172.21.15.86:0/3096850506 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fba280ac840 con 0x7fba280b1f60 2024-03-29T11:27:42.753 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.757+0000 7fba24ff9640 1 -- 172.21.15.86:0/3096850506 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fba18001bf0 con 0x7fba280b1f60 2024-03-29T11:27:42.753 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.757+0000 7fba24ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:42.754 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.757+0000 7fba24ff9640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:42.754 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.757+0000 7fba24ff9640 10 monclient(hunting): dump: 2024-03-29T11:27:42.754 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:42.754 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:42.754 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:42.754 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:42.754 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:42.754 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:42.754 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:42.754 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:42.754 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:42.754 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:42.754 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.757+0000 7fba24ff9640 1 -- 172.21.15.86:0/3096850506 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1506509858 0 0) 0x7fba14005300 con 0x7fba28143d10 2024-03-29T11:27:42.754 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.757+0000 7fba24ff9640 1 -- 172.21.15.86:0/3096850506 >> v1:172.21.15.196:6789/0 conn(0x7fba280aa530 legacy=0x7fba281e7940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:42.754 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.757+0000 7fba24ff9640 1 -- 172.21.15.86:0/3096850506 >> v1:172.21.15.86:6789/0 conn(0x7fba280b1f60 legacy=0x7fba2813fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:42.755 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.757+0000 7fba24ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:27:42.755 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.757+0000 7fba24ff9640 1 monclient: found mon.c 2024-03-29T11:27:42.755 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.757+0000 7fba24ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:42.755 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.757+0000 7fba24ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:27:42.755 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.757+0000 7fba24ff9640 1 -- 172.21.15.86:0/3096850506 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba280aaed0 con 0x7fba28143d10 2024-03-29T11:27:42.755 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.757+0000 7fba24ff9640 10 monclient: _finish_auth 0 2024-03-29T11:27:42.755 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.757+0000 7fba24ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:27:42.755 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.757+0000 7fba24ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:42.755 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.758+0000 7fba2e7e2640 5 monclient: authenticate success, global_id 4529 2024-03-29T11:27:42.755 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.758+0000 7fba2e7e2640 10 monclient: _renew_subs 2024-03-29T11:27:42.755 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.758+0000 7fba2e7e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:27:42.755 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.758+0000 7fba2e7e2640 1 -- 172.21.15.86:0/3096850506 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba28154e90 con 0x7fba28143d10 2024-03-29T11:27:42.755 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.758+0000 7fba2e7e2640 10 client.4529.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:42.755 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.758+0000 7fba2e7e2640 10 monclient: _renew_subs 2024-03-29T11:27:42.756 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.758+0000 7fba2e7e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:27:42.756 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.758+0000 7fba2e7e2640 1 -- 172.21.15.86:0/3096850506 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fba281403d0 con 0x7fba28143d10 2024-03-29T11:27:42.756 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.758+0000 7fba2e7e2640 1 librados: init done 2024-03-29T11:27:42.756 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.759+0000 7fba24ff9640 1 -- 172.21.15.86:0/3096850506 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fba1c001cb0 con 0x7fba28143d10 2024-03-29T11:27:42.756 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.759+0000 7fba24ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:42.756 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.759+0000 7fba24ff9640 1 -- 172.21.15.86:0/3096850506 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fba1c003a60 con 0x7fba28143d10 2024-03-29T11:27:42.756 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.759+0000 7fba24ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:42.756 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.759+0000 7fba24ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:42.756 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.759+0000 7fba2e7e2640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:42.756 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.759+0000 7fba24ff9640 10 monclient: dump: 2024-03-29T11:27:42.756 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:42.756 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:42.756 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:42.756 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:42.756 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:42.757 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:42.757 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:42.757 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:42.757 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:42.757 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:42.757 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.759+0000 7fba267fc640 4 set_mon_vals no callback set 2024-03-29T11:27:42.757 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.761+0000 7fba2e7e2640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:42.757 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.761+0000 7fba2e7e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:27:42.757 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.761+0000 7fba2e7e2640 1 -- 172.21.15.86:0/3096850506 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba2806ef80 con 0x7fba28143d10 2024-03-29T11:27:42.757 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.761+0000 7fba24ff9640 1 -- 172.21.15.86:0/3096850506 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7fba1c003c80 con 0x7fba28143d10 2024-03-29T11:27:42.757 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.762+0000 7fba24ff9640 1 -- 172.21.15.86:0/3096850506 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7fba1c084a80 con 0x7fba28143d10 2024-03-29T11:27:42.757 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.762+0000 7fba24ff9640 10 client.4529.objecter ms_dispatch 0x7fba28000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:42.757 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.762+0000 7fba24ff9640 3 client.4529.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:42.757 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.762+0000 7fba24ff9640 3 client.4529.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:42.758 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.762+0000 7fba24ff9640 20 client.4529.objecter dump_active .. 0 homeless 2024-03-29T11:27:42.758 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.763+0000 7fba27fff640 10 monclient: get_auth_request con 0x7fba1406e0b0 auth_method 0 2024-03-29T11:27:42.758 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.765+0000 7fba24ff9640 10 client.4529.objecter ms_handle_connect 0x7fba1406e0b0 2024-03-29T11:27:42.758 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.769+0000 7fba24ff9640 1 -- 172.21.15.86:0/3096850506 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fba1c0503c0 con 0x7fba28143d10 2024-03-29T11:27:42.758 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.769+0000 7fba24ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:42.759 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:42.769+0000 7fba24ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:43.095 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:43.103+0000 7fba2e7e2640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:43.095 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:43.103+0000 7fba2e7e2640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:43.095 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:43.103+0000 7fba2e7e2640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:27:43.095 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:43.103+0000 7fba2e7e2640 1 -- 172.21.15.86:0/3096850506 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fba280ab060 con 0x7fba28143d10 2024-03-29T11:27:43.095 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:43.104+0000 7fba24ff9640 1 -- 172.21.15.86:0/3096850506 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7fba280ab060 con 0x7fba28143d10 2024-03-29T11:27:43.096 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:43.104+0000 7fba24ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:43.096 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:43.104+0000 7fba24ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:43.096 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:27:43.096 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:43.097 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:43.108+0000 7fba2e7e2640 10 librados: watch_flush enter 2024-03-29T11:27:43.097 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:43.108+0000 7fba2e7e2640 10 librados: watch_flush exit 2024-03-29T11:27:43.097 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:43.108+0000 7fba2e7e2640 20 client.4529.objecter shutdown clearing up homeless session... 2024-03-29T11:27:43.097 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:43.108+0000 7fba2e7e2640 10 client.4529.objecter successfully canceled tick 2024-03-29T11:27:43.098 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:43.108+0000 7fba2e7e2640 1 -- 172.21.15.86:0/3096850506 >> v1:172.21.15.196:6816/588110318 conn(0x7fba1406e0b0 legacy=0x7fba14070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:43.098 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:43.108+0000 7fba2e7e2640 10 monclient: shutdown 2024-03-29T11:27:43.098 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:43.108+0000 7fba2e7e2640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:43.098 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:43.108+0000 7fba2e7e2640 1 -- 172.21.15.86:0/3096850506 >> v1:172.21.15.86:6790/0 conn(0x7fba28143d10 legacy=0x7fba281e7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:43.098 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:43.108+0000 7fba2e7e2640 1 -- 172.21.15.86:0/3096850506 shutdown_connections 2024-03-29T11:27:43.098 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:43.108+0000 7fba2e7e2640 1 -- 172.21.15.86:0/3096850506 >> 172.21.15.86:0/3096850506 conn(0x7fba28066a00 msgr2=0x7fba280b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:43.098 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:43.108+0000 7fba2e7e2640 1 -- 172.21.15.86:0/3096850506 shutdown_connections 2024-03-29T11:27:43.098 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:43.109+0000 7fba2e7e2640 1 -- 172.21.15.86:0/3096850506 wait complete. 2024-03-29T11:27:43.098 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:43.109+0000 7fba2e7e2640 1 librados: shutdown 2024-03-29T11:27:43.115 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:49.921 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:50.068 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.079+0000 7fd9aacba640 10 monclient: get_monmap_and_config 2024-03-29T11:27:50.068 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.079+0000 7fd9aacba640 10 monclient: build_initial_monmap 2024-03-29T11:27:50.069 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.079+0000 7fd9aacba640 10 monclient: monmap: 2024-03-29T11:27:50.069 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:27:50.069 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:50.069 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:27:50.080231+0000 2024-03-29T11:27:50.069 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:27:50.080231+0000 2024-03-29T11:27:50.069 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:50.069 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:50.069 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:27:50.069 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:27:50.069 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:27:50.069 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:50.069 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.080+0000 7fd9aacba640 1 Processor -- start 2024-03-29T11:27:50.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.080+0000 7fd9aacba640 1 -- start start 2024-03-29T11:27:50.070 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.080+0000 7fd9aacba640 10 monclient: init 2024-03-29T11:27:50.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.081+0000 7fd9aacba640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:50.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.081+0000 7fd9aacba640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:50.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.081+0000 7fd9aacba640 10 monclient(hunting): picked mon.noname-a con 0x7fd99c006800 addr v1:172.21.15.86:6789/0 2024-03-29T11:27:50.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.081+0000 7fd9aacba640 10 monclient(hunting): picked mon.noname-c con 0x7fd99c006be0 addr v1:172.21.15.196:6789/0 2024-03-29T11:27:50.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.081+0000 7fd9aacba640 10 monclient(hunting): picked mon.noname-b con 0x7fd99c0c95e0 addr v1:172.21.15.86:6790/0 2024-03-29T11:27:50.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.081+0000 7fd9aacba640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd99c0c5570 con 0x7fd99c006800 2024-03-29T11:27:50.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.081+0000 7fd9aacba640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd99c02e450 con 0x7fd99c006be0 2024-03-29T11:27:50.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.081+0000 7fd9aacba640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd99c0c31c0 con 0x7fd99c0c95e0 2024-03-29T11:27:50.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.081+0000 7fd9aacba640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:50.071 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.081+0000 7fd9aacba640 10 monclient(hunting): authenticate will time out at 1294.399617s 2024-03-29T11:27:50.074 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.085+0000 7fd9aa4b9640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7fd99c0c95e0 0x7fd99c0d0e90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:39969/0 (socket says 172.21.15.86:39969) 2024-03-29T11:27:50.075 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.085+0000 7fd9aa4b9640 1 -- 172.21.15.86:0/1543546669 learned_addr learned my addr 172.21.15.86:0/1543546669 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:27:50.075 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.085+0000 7fd9a8cb6640 1 -- 172.21.15.86:0/1543546669 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4161597909 0 0) 0x7fd99c0c31c0 con 0x7fd99c0c95e0 2024-03-29T11:27:50.075 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.085+0000 7fd9a8cb6640 10 monclient(hunting): _negotiate 2024-03-29T11:27:50.075 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.085+0000 7fd9a8cb6640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:50.075 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.085+0000 7fd9a8cb6640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:50.075 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.085+0000 7fd9a8cb6640 10 monclient(hunting): my global_id is 4532 2024-03-29T11:27:50.075 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.085+0000 7fd9a8cb6640 1 -- 172.21.15.86:0/1543546669 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd990003520 con 0x7fd99c0c95e0 2024-03-29T11:27:50.075 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.086+0000 7fd9a8cb6640 1 -- 172.21.15.86:0/1543546669 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 81793461 0 0) 0x7fd990003520 con 0x7fd99c0c95e0 2024-03-29T11:27:50.075 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.086+0000 7fd9a8cb6640 1 -- 172.21.15.86:0/1543546669 >> v1:172.21.15.196:6789/0 conn(0x7fd99c006be0 legacy=0x7fd99c0c8ed0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:27:50.077 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.088+0000 7fd9a8cb6640 1 -- 172.21.15.86:0/1543546669 >> v1:172.21.15.86:6789/0 conn(0x7fd99c006800 legacy=0x7fd99c0c2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:50.079 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.089+0000 7fd9a8cb6640 10 monclient: _finish_hunting 0 2024-03-29T11:27:50.079 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.089+0000 7fd9a8cb6640 1 monclient: found mon.noname-b 2024-03-29T11:27:50.079 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.089+0000 7fd9a8cb6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:50.079 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.089+0000 7fd9a8cb6640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:27:50.079 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.089+0000 7fd9a8cb6640 1 -- 172.21.15.86:0/1543546669 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd99c02cde0 con 0x7fd99c0c95e0 2024-03-29T11:27:50.079 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.089+0000 7fd9a8cb6640 10 monclient: _finish_auth 0 2024-03-29T11:27:50.079 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.089+0000 7fd9a8cb6640 10 monclient: _check_auth_tickets 2024-03-29T11:27:50.079 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.089+0000 7fd9a8cb6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:50.079 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.089+0000 7fd9a8cb6640 1 -- 172.21.15.86:0/1543546669 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fd9a404f730 con 0x7fd99c0c95e0 2024-03-29T11:27:50.079 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.089+0000 7fd9a8cb6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:50.080 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.089+0000 7fd9a8cb6640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:50.080 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.089+0000 7fd9a8cb6640 10 monclient: dump: 2024-03-29T11:27:50.080 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:50.080 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:50.080 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:50.080 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:50.080 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:50.080 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:50.080 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:50.080 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:50.080 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:50.081 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:50.081 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.089+0000 7fd9aacba640 5 monclient: authenticate success, global_id 4532 2024-03-29T11:27:50.081 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.089+0000 7fd9aacba640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:50.081 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.089+0000 7fd9a8cb6640 1 -- 172.21.15.86:0/1543546669 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd9a40008d0 con 0x7fd99c0c95e0 2024-03-29T11:27:50.081 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.089+0000 7fd9a8cb6640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:50.081 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.089+0000 7fd9aacba640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:50.081 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.089+0000 7fd9aacba640 4 set_mon_vals no callback set 2024-03-29T11:27:50.081 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.090+0000 7fd9aacba640 10 monclient: shutdown 2024-03-29T11:27:50.081 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.090+0000 7fd9aacba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:50.081 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.090+0000 7fd9aacba640 1 -- 172.21.15.86:0/1543546669 >> v1:172.21.15.86:6790/0 conn(0x7fd99c0c95e0 legacy=0x7fd99c0d0e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:50.081 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.090+0000 7fd9aacba640 1 -- 172.21.15.86:0/1543546669 shutdown_connections 2024-03-29T11:27:50.081 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.090+0000 7fd9aacba640 1 -- 172.21.15.86:0/1543546669 >> 172.21.15.86:0/1543546669 conn(0x7fd99c0058f0 msgr2=0x7fd99c005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:50.082 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.092+0000 7fd9aacba640 1 -- 172.21.15.86:0/1543546669 shutdown_connections 2024-03-29T11:27:50.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.093+0000 7fd9aacba640 1 -- 172.21.15.86:0/1543546669 wait complete. 2024-03-29T11:27:50.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.093+0000 7fd9aacba640 10 monclient: build_initial_monmap 2024-03-29T11:27:50.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.093+0000 7fd9aacba640 10 monclient: monmap: 2024-03-29T11:27:50.084 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:27:50.084 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:50.084 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:27:50.085 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:27:50.085 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:50.085 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:50.085 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:27:50.085 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:27:50.085 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:27:50.085 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:50.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.094+0000 7fd9aacba640 1 librados: starting msgr at 2024-03-29T11:27:50.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.094+0000 7fd9aacba640 1 librados: starting objecter 2024-03-29T11:27:50.085 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.094+0000 7fd9aacba640 1 Processor -- start 2024-03-29T11:27:50.086 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.094+0000 7fd9aacba640 1 -- start start 2024-03-29T11:27:50.086 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.094+0000 7fd9aacba640 1 librados: setting wanted keys 2024-03-29T11:27:50.086 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.094+0000 7fd9aacba640 1 librados: calling monclient init 2024-03-29T11:27:50.086 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.094+0000 7fd9aacba640 10 monclient: init 2024-03-29T11:27:50.086 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.095+0000 7fd9aacba640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:50.088 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.095+0000 7fd9aacba640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:27:50.089 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.095+0000 7fd9aacba640 10 monclient(hunting): picked mon.noname-b con 0x7fd99c006800 addr v1:172.21.15.196:6789/0 2024-03-29T11:27:50.089 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.095+0000 7fd9aacba640 10 monclient(hunting): picked mon.noname-c con 0x7fd99c0c95e0 addr v1:172.21.15.86:6790/0 2024-03-29T11:27:50.089 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.095+0000 7fd9aacba640 10 monclient(hunting): picked mon.noname-a con 0x7fd99c16d5b0 addr v1:172.21.15.86:6789/0 2024-03-29T11:27:50.089 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.095+0000 7fd9aacba640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd99c16d120 con 0x7fd99c16d5b0 2024-03-29T11:27:50.089 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.095+0000 7fd9aacba640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd99c0c5460 con 0x7fd99c006800 2024-03-29T11:27:50.089 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.095+0000 7fd9aacba640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd99c169bd0 con 0x7fd99c0c95e0 2024-03-29T11:27:50.089 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.095+0000 7fd9aacba640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:50.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.095+0000 7fd9aacba640 10 monclient(hunting): authenticate will time out at 1294.413375s 2024-03-29T11:27:50.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.099+0000 7fd9a9cb8640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fd99c006800 0x7fd99c1694c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:52909/0 (socket says 172.21.15.86:52909) 2024-03-29T11:27:50.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.099+0000 7fd9a9cb8640 1 -- 172.21.15.86:0/1329686179 learned_addr learned my addr 172.21.15.86:0/1329686179 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:27:50.094 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.102+0000 7fd99a7fc640 10 client.?.objecter ms_handle_connect 0x7fd99c0c95e0 2024-03-29T11:27:50.095 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.102+0000 7fd99a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:50.095 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.102+0000 7fd99a7fc640 1 -- 172.21.15.86:0/1329686179 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4229164978 0 0) 0x7fd99c169bd0 con 0x7fd99c0c95e0 2024-03-29T11:27:50.095 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.102+0000 7fd99a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:50.095 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.102+0000 7fd99a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:50.095 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.102+0000 7fd99a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:50.095 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.102+0000 7fd99a7fc640 10 monclient(hunting): my global_id is 4535 2024-03-29T11:27:50.095 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.102+0000 7fd99a7fc640 1 -- 172.21.15.86:0/1329686179 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd988004eb0 con 0x7fd99c0c95e0 2024-03-29T11:27:50.095 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.102+0000 7fd99a7fc640 10 client.?.objecter ms_handle_connect 0x7fd99c006800 2024-03-29T11:27:50.096 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.102+0000 7fd99a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:50.096 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.102+0000 7fd99a7fc640 1 -- 172.21.15.86:0/1329686179 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2678671903 0 0) 0x7fd99c0c5460 con 0x7fd99c006800 2024-03-29T11:27:50.096 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.102+0000 7fd99a7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:50.096 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.102+0000 7fd99a7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:50.096 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.102+0000 7fd99a7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:50.096 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.103+0000 7fd99a7fc640 10 monclient(hunting): my global_id is 4450 2024-03-29T11:27:50.096 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.103+0000 7fd99a7fc640 1 -- 172.21.15.86:0/1329686179 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd99c169bd0 con 0x7fd99c006800 2024-03-29T11:27:50.096 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.105+0000 7fd99a7fc640 1 -- 172.21.15.86:0/1329686179 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3431890909 0 0) 0x7fd99c169bd0 con 0x7fd99c006800 2024-03-29T11:27:50.097 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.105+0000 7fd99a7fc640 1 -- 172.21.15.86:0/1329686179 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd99c0c5460 con 0x7fd99c006800 2024-03-29T11:27:50.097 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.105+0000 7fd99a7fc640 1 -- 172.21.15.86:0/1329686179 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fd994003fa0 con 0x7fd99c006800 2024-03-29T11:27:50.097 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.105+0000 7fd99a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:50.097 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.105+0000 7fd99a7fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:27:50.097 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.105+0000 7fd99a7fc640 10 monclient(hunting): dump: 2024-03-29T11:27:50.097 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:50.097 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:50.097 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:50.097 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:50.098 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:50.098 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:50.098 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:50.098 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:50.098 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:50.098 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:50.098 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.105+0000 7fd99a7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:50.099 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.105+0000 7fd99a7fc640 1 -- 172.21.15.86:0/1329686179 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 135233031 0 0) 0x7fd988004eb0 con 0x7fd99c0c95e0 2024-03-29T11:27:50.099 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.105+0000 7fd99a7fc640 1 -- 172.21.15.86:0/1329686179 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fd99c169bd0 con 0x7fd99c0c95e0 2024-03-29T11:27:50.099 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.105+0000 7fd99a7fc640 1 -- 172.21.15.86:0/1329686179 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fd9a0001870 con 0x7fd99c0c95e0 2024-03-29T11:27:50.099 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.105+0000 7fd99a7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:50.099 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.105+0000 7fd99a7fc640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:50.099 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.105+0000 7fd99a7fc640 10 monclient(hunting): dump: 2024-03-29T11:27:50.099 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:50.099 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:50.100 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:50.100 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:50.100 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:50.100 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:50.100 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:50.100 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:50.100 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:50.100 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:50.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.106+0000 7fd99a7fc640 10 client.?.objecter ms_handle_connect 0x7fd99c16d5b0 2024-03-29T11:27:50.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.106+0000 7fd99a7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:50.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.107+0000 7fd99a7fc640 1 -- 172.21.15.86:0/1329686179 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3040417193 0 0) 0x7fd99c0c5460 con 0x7fd99c006800 2024-03-29T11:27:50.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.107+0000 7fd99a7fc640 1 -- 172.21.15.86:0/1329686179 >> v1:172.21.15.86:6790/0 conn(0x7fd99c0c95e0 legacy=0x7fd99c0c1cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:50.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.107+0000 7fd99a7fc640 1 -- 172.21.15.86:0/1329686179 >> v1:172.21.15.86:6789/0 conn(0x7fd99c16d5b0 legacy=0x7fd99c174d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:50.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.107+0000 7fd99a7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:27:50.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.107+0000 7fd99a7fc640 1 monclient: found mon.b 2024-03-29T11:27:50.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.107+0000 7fd99a7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:50.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.109+0000 7fd99a7fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:27:50.110 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.110+0000 7fd99a7fc640 1 -- 172.21.15.86:0/1329686179 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd99c032b00 con 0x7fd99c006800 2024-03-29T11:27:50.110 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.110+0000 7fd99a7fc640 10 monclient: _finish_auth 0 2024-03-29T11:27:50.110 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.110+0000 7fd99a7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:27:50.110 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.110+0000 7fd99a7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:50.110 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.110+0000 7fd9aacba640 5 monclient: authenticate success, global_id 4450 2024-03-29T11:27:50.110 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.110+0000 7fd9aacba640 10 monclient: _renew_subs 2024-03-29T11:27:50.110 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.110+0000 7fd9aacba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:27:50.110 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.110+0000 7fd9aacba640 1 -- 172.21.15.86:0/1329686179 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd99c15f8b0 con 0x7fd99c006800 2024-03-29T11:27:50.111 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.110+0000 7fd9aacba640 10 client.4450.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:50.111 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.110+0000 7fd9aacba640 10 monclient: _renew_subs 2024-03-29T11:27:50.111 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.110+0000 7fd9aacba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:27:50.111 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.110+0000 7fd9aacba640 1 -- 172.21.15.86:0/1329686179 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fd99c0c25e0 con 0x7fd99c006800 2024-03-29T11:27:50.111 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.110+0000 7fd9aacba640 1 librados: init done 2024-03-29T11:27:50.111 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.111+0000 7fd99a7fc640 1 -- 172.21.15.86:0/1329686179 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fd994004100 con 0x7fd99c006800 2024-03-29T11:27:50.111 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.111+0000 7fd99a7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:50.111 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.111+0000 7fd99a7fc640 1 -- 172.21.15.86:0/1329686179 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fd994001070 con 0x7fd99c006800 2024-03-29T11:27:50.112 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.111+0000 7fd99a7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:50.112 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.111+0000 7fd99a7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:27:50.112 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.111+0000 7fd99a7fc640 10 monclient: dump: 2024-03-29T11:27:50.112 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:50.112 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:50.112 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:50.112 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:50.112 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:50.113 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:50.113 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:50.114 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:50.114 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:50.114 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:50.114 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.113+0000 7fd9aacba640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:50.114 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.113+0000 7fd99bfff640 4 set_mon_vals no callback set 2024-03-29T11:27:50.114 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.114+0000 7fd9aacba640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:50.115 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.114+0000 7fd9aacba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:27:50.115 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.114+0000 7fd9aacba640 1 -- 172.21.15.86:0/1329686179 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd99c15fc50 con 0x7fd99c006800 2024-03-29T11:27:50.115 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.117+0000 7fd99a7fc640 1 -- 172.21.15.86:0/1329686179 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7fd9940012d0 con 0x7fd99c006800 2024-03-29T11:27:50.115 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.118+0000 7fd99a7fc640 1 -- 172.21.15.86:0/1329686179 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7fd994084980 con 0x7fd99c006800 2024-03-29T11:27:50.115 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.118+0000 7fd99a7fc640 10 client.4450.objecter ms_dispatch 0x7fd9a4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:50.115 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.118+0000 7fd99a7fc640 3 client.4450.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:50.115 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.118+0000 7fd9a94b7640 10 monclient: get_auth_request con 0x7fd98806dca0 auth_method 0 2024-03-29T11:27:50.115 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.118+0000 7fd99a7fc640 3 client.4450.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:50.115 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.119+0000 7fd99a7fc640 20 client.4450.objecter dump_active .. 0 homeless 2024-03-29T11:27:50.115 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.119+0000 7fd99a7fc640 10 client.4450.objecter ms_handle_connect 0x7fd98806dca0 2024-03-29T11:27:50.116 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.123+0000 7fd99a7fc640 1 -- 172.21.15.86:0/1329686179 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fd994050260 con 0x7fd99c006800 2024-03-29T11:27:50.116 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.123+0000 7fd99a7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:50.117 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.123+0000 7fd99a7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:50.431 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.442+0000 7fd9aacba640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:50.431 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.442+0000 7fd9aacba640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:50.431 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.442+0000 7fd9aacba640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:27:50.431 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.442+0000 7fd9aacba640 1 -- 172.21.15.86:0/1329686179 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd99c004cd0 con 0x7fd99c006800 2024-03-29T11:27:50.433 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.445+0000 7fd99a7fc640 1 -- 172.21.15.86:0/1329686179 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7fd994052190 con 0x7fd99c006800 2024-03-29T11:27:50.434 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:27:50.434 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:50.436 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.445+0000 7fd99a7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:50.436 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.445+0000 7fd99a7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:50.437 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.449+0000 7fd9aacba640 10 librados: watch_flush enter 2024-03-29T11:27:50.438 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.449+0000 7fd9aacba640 10 librados: watch_flush exit 2024-03-29T11:27:50.438 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.449+0000 7fd9aacba640 20 client.4450.objecter shutdown clearing up homeless session... 2024-03-29T11:27:50.438 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.449+0000 7fd9aacba640 10 client.4450.objecter successfully canceled tick 2024-03-29T11:27:50.438 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.449+0000 7fd9aacba640 1 -- 172.21.15.86:0/1329686179 >> v1:172.21.15.196:6816/588110318 conn(0x7fd98806dca0 legacy=0x7fd988070160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:50.438 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.449+0000 7fd9aacba640 10 monclient: shutdown 2024-03-29T11:27:50.438 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.449+0000 7fd9aacba640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:50.438 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.449+0000 7fd9aacba640 1 -- 172.21.15.86:0/1329686179 >> v1:172.21.15.196:6789/0 conn(0x7fd99c006800 legacy=0x7fd99c1694c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:50.438 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.449+0000 7fd9aacba640 1 -- 172.21.15.86:0/1329686179 shutdown_connections 2024-03-29T11:27:50.438 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.449+0000 7fd9aacba640 1 -- 172.21.15.86:0/1329686179 >> 172.21.15.86:0/1329686179 conn(0x7fd99c0058f0 msgr2=0x7fd99c0050f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:50.439 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.450+0000 7fd9aacba640 1 -- 172.21.15.86:0/1329686179 shutdown_connections 2024-03-29T11:27:50.439 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.450+0000 7fd9aacba640 1 -- 172.21.15.86:0/1329686179 wait complete. 2024-03-29T11:27:50.439 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:50.450+0000 7fd9aacba640 1 librados: shutdown 2024-03-29T11:27:50.459 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:27:57.266 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:27:57.417 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.428+0000 7f2ae7cb3640 10 monclient: get_monmap_and_config 2024-03-29T11:27:57.417 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.428+0000 7f2ae7cb3640 10 monclient: build_initial_monmap 2024-03-29T11:27:57.418 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.428+0000 7f2ae7cb3640 10 monclient: monmap: 2024-03-29T11:27:57.418 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:27:57.418 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:57.418 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:27:57.429708+0000 2024-03-29T11:27:57.418 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:27:57.429708+0000 2024-03-29T11:27:57.418 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:57.418 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:57.418 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:27:57.418 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:27:57.418 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:27:57.418 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:57.419 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.429+0000 7f2ae7cb3640 1 Processor -- start 2024-03-29T11:27:57.419 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.430+0000 7f2ae7cb3640 1 -- start start 2024-03-29T11:27:57.419 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.430+0000 7f2ae7cb3640 10 monclient: init 2024-03-29T11:27:57.420 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.431+0000 7f2ae7cb3640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:57.420 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.431+0000 7f2ae7cb3640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:27:57.422 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.431+0000 7f2ae7cb3640 10 monclient(hunting): picked mon.noname-a con 0x7f2ae01422b0 addr v1:172.21.15.86:6789/0 2024-03-29T11:27:57.422 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.431+0000 7f2ae7cb3640 10 monclient(hunting): picked mon.noname-c con 0x7f2ae0142690 addr v1:172.21.15.196:6789/0 2024-03-29T11:27:57.422 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.431+0000 7f2ae7cb3640 10 monclient(hunting): picked mon.noname-b con 0x7f2ae00b17d0 addr v1:172.21.15.86:6790/0 2024-03-29T11:27:57.422 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.431+0000 7f2ae7cb3640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ae0142c80 con 0x7f2ae01422b0 2024-03-29T11:27:57.422 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.431+0000 7f2ae7cb3640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ae00aae40 con 0x7f2ae0142690 2024-03-29T11:27:57.422 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.431+0000 7f2ae7cb3640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ae013fbd0 con 0x7f2ae00b17d0 2024-03-29T11:27:57.422 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.431+0000 7f2ae7cb3640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:57.422 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.431+0000 7f2ae7cb3640 10 monclient(hunting): authenticate will time out at 1301.749433s 2024-03-29T11:27:57.422 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.431+0000 7f2ae5a4f640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f2ae01422b0 0x7f2ae013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:44020/0 (socket says 172.21.15.86:44020) 2024-03-29T11:27:57.423 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.431+0000 7f2ae5a4f640 1 -- 172.21.15.86:0/4211518886 learned_addr learned my addr 172.21.15.86:0/4211518886 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:27:57.423 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.432+0000 7f2ae4a4d640 1 -- 172.21.15.86:0/4211518886 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2573477363 0 0) 0x7f2ae0142c80 con 0x7f2ae01422b0 2024-03-29T11:27:57.423 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.432+0000 7f2ae4a4d640 10 monclient(hunting): _negotiate 2024-03-29T11:27:57.423 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.432+0000 7f2ae4a4d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:57.423 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.432+0000 7f2ae4a4d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:57.423 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.432+0000 7f2ae4a4d640 10 monclient(hunting): my global_id is 4464 2024-03-29T11:27:57.423 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.432+0000 7f2ae4a4d640 1 -- 172.21.15.86:0/4211518886 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2acc003520 con 0x7f2ae01422b0 2024-03-29T11:27:57.423 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.432+0000 7f2ae4a4d640 1 -- 172.21.15.86:0/4211518886 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1643852655 0 0) 0x7f2ae013fbd0 con 0x7f2ae00b17d0 2024-03-29T11:27:57.423 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.432+0000 7f2ae4a4d640 10 monclient(hunting): _negotiate 2024-03-29T11:27:57.423 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.432+0000 7f2ae4a4d640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:57.423 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.432+0000 7f2ae4a4d640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:57.423 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.432+0000 7f2ae4a4d640 10 monclient(hunting): my global_id is 4538 2024-03-29T11:27:57.424 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.432+0000 7f2ae4a4d640 1 -- 172.21.15.86:0/4211518886 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ae0142c80 con 0x7f2ae00b17d0 2024-03-29T11:27:57.424 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.432+0000 7f2ae4a4d640 1 -- 172.21.15.86:0/4211518886 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3873636272 0 0) 0x7f2acc003520 con 0x7f2ae01422b0 2024-03-29T11:27:57.424 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.432+0000 7f2ae4a4d640 1 -- 172.21.15.86:0/4211518886 >> v1:172.21.15.86:6790/0 conn(0x7f2ae00b17d0 legacy=0x7f2ae00b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:57.424 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.432+0000 7f2ae4a4d640 1 -- 172.21.15.86:0/4211518886 >> v1:172.21.15.196:6789/0 conn(0x7f2ae0142690 legacy=0x7f2ae00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:27:57.424 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.433+0000 7f2ae4a4d640 10 monclient: _finish_hunting 0 2024-03-29T11:27:57.424 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.433+0000 7f2ae4a4d640 1 monclient: found mon.noname-a 2024-03-29T11:27:57.424 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.433+0000 7f2ae4a4d640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:57.424 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.433+0000 7f2ae4a4d640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.86:6789/0 2024-03-29T11:27:57.424 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.433+0000 7f2ae4a4d640 1 -- 172.21.15.86:0/4211518886 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae0151940 con 0x7f2ae01422b0 2024-03-29T11:27:57.424 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.433+0000 7f2ae4a4d640 10 monclient: _finish_auth 0 2024-03-29T11:27:57.424 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.433+0000 7f2ae4a4d640 10 monclient: _check_auth_tickets 2024-03-29T11:27:57.424 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.433+0000 7f2ae4a4d640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:57.425 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.433+0000 7f2ae4a4d640 1 -- 172.21.15.86:0/4211518886 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f2ad8001a10 con 0x7f2ae01422b0 2024-03-29T11:27:57.425 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.433+0000 7f2ae4a4d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:57.425 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.433+0000 7f2ae4a4d640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:27:57.425 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.433+0000 7f2ae4a4d640 10 monclient: dump: 2024-03-29T11:27:57.425 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:57.425 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:57.425 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:57.425 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:57.425 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:57.425 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:57.425 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:57.425 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:57.425 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:57.425 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:57.426 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.433+0000 7f2ae7cb3640 5 monclient: authenticate success, global_id 4464 2024-03-29T11:27:57.426 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.433+0000 7f2ae7cb3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:27:57.433 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.433+0000 7f2ae4a4d640 1 -- 172.21.15.86:0/4211518886 <== mon.0 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ad8001b70 con 0x7f2ae01422b0 2024-03-29T11:27:57.433 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.433+0000 7f2ae4a4d640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:57.433 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.433+0000 7f2ae4a4d640 1 -- 172.21.15.86:0/4211518886 <== mon.0 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f2ad8001f00 con 0x7f2ae01422b0 2024-03-29T11:27:57.433 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.433+0000 7f2ae4a4d640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:57.433 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.433+0000 7f2ae4a4d640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:27:57.433 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.433+0000 7f2ae4a4d640 10 monclient: dump: 2024-03-29T11:27:57.433 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:57.433 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:57.433 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:57.433 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:57.434 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:57.434 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:57.434 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:57.434 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:57.434 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:57.434 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:57.434 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.433+0000 7f2ae7cb3640 10 monclient: get_monmap_and_config success 2024-03-29T11:27:57.434 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.433+0000 7f2ae7cb3640 4 set_mon_vals no callback set 2024-03-29T11:27:57.434 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.434+0000 7f2ae7cb3640 10 monclient: shutdown 2024-03-29T11:27:57.434 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.434+0000 7f2ae7cb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:57.434 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.434+0000 7f2ae7cb3640 1 -- 172.21.15.86:0/4211518886 >> v1:172.21.15.86:6789/0 conn(0x7f2ae01422b0 legacy=0x7f2ae013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:57.434 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.434+0000 7f2ae7cb3640 1 -- 172.21.15.86:0/4211518886 shutdown_connections 2024-03-29T11:27:57.434 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.434+0000 7f2ae7cb3640 1 -- 172.21.15.86:0/4211518886 >> 172.21.15.86:0/4211518886 conn(0x7f2ae0066a00 msgr2=0x7f2ae0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:57.434 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.434+0000 7f2ae7cb3640 1 -- 172.21.15.86:0/4211518886 shutdown_connections 2024-03-29T11:27:57.435 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.434+0000 7f2ae7cb3640 1 -- 172.21.15.86:0/4211518886 wait complete. 2024-03-29T11:27:57.435 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.435+0000 7f2ae7cb3640 10 monclient: build_initial_monmap 2024-03-29T11:27:57.435 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.436+0000 7f2ae7cb3640 10 monclient: monmap: 2024-03-29T11:27:57.435 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:27:57.435 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:27:57.435 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:27:57.435 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:27:57.435 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:27:57.435 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:57.435 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:27:57.435 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:27:57.435 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:27:57.437 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:57.437 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.436+0000 7f2ae7cb3640 1 librados: starting msgr at 2024-03-29T11:27:57.437 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.436+0000 7f2ae7cb3640 1 librados: starting objecter 2024-03-29T11:27:57.437 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.436+0000 7f2ae7cb3640 1 Processor -- start 2024-03-29T11:27:57.437 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.436+0000 7f2ae7cb3640 1 -- start start 2024-03-29T11:27:57.438 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.436+0000 7f2ae7cb3640 1 librados: setting wanted keys 2024-03-29T11:27:57.438 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.436+0000 7f2ae7cb3640 1 librados: calling monclient init 2024-03-29T11:27:57.438 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.436+0000 7f2ae7cb3640 10 monclient: init 2024-03-29T11:27:57.438 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.437+0000 7f2ae7cb3640 10 monclient: _reopen_session rank -1 2024-03-29T11:27:57.438 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.437+0000 7f2ae7cb3640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:27:57.438 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.437+0000 7f2ae7cb3640 10 monclient(hunting): picked mon.noname-c con 0x7f2ae00b17d0 addr v1:172.21.15.86:6790/0 2024-03-29T11:27:57.438 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.437+0000 7f2ae7cb3640 10 monclient(hunting): picked mon.noname-a con 0x7f2ae01422b0 addr v1:172.21.15.86:6789/0 2024-03-29T11:27:57.438 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.437+0000 7f2ae7cb3640 10 monclient(hunting): picked mon.noname-b con 0x7f2ae0142690 addr v1:172.21.15.196:6789/0 2024-03-29T11:27:57.438 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.437+0000 7f2ae7cb3640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ae006faa0 con 0x7f2ae01422b0 2024-03-29T11:27:57.438 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.437+0000 7f2ae7cb3640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ae00ab670 con 0x7f2ae0142690 2024-03-29T11:27:57.438 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.437+0000 7f2ae7cb3640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ae00ab950 con 0x7f2ae00b17d0 2024-03-29T11:27:57.438 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.437+0000 7f2ae7cb3640 10 monclient(hunting): _renew_subs 2024-03-29T11:27:57.438 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.437+0000 7f2ae7cb3640 10 monclient(hunting): authenticate will time out at 1301.755775s 2024-03-29T11:27:57.439 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.438+0000 7f2ae5a4f640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f2ae00b17d0 0x7f2ae0070d30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:46308/0 (socket says 172.21.15.86:46308) 2024-03-29T11:27:57.439 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.438+0000 7f2ae5a4f640 1 -- 172.21.15.86:0/3692025930 learned_addr learned my addr 172.21.15.86:0/3692025930 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:27:57.439 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.438+0000 7f2ad67fc640 10 client.?.objecter ms_handle_connect 0x7f2ae00b17d0 2024-03-29T11:27:57.439 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.438+0000 7f2ad67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:57.439 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.438+0000 7f2ad67fc640 1 -- 172.21.15.86:0/3692025930 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1735148367 0 0) 0x7f2ae00ab950 con 0x7f2ae00b17d0 2024-03-29T11:27:57.439 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.438+0000 7f2ad67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:57.439 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.438+0000 7f2ad67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:57.439 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.438+0000 7f2ad67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:57.439 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.438+0000 7f2ad67fc640 10 monclient(hunting): my global_id is 4541 2024-03-29T11:27:57.439 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.438+0000 7f2ad67fc640 1 -- 172.21.15.86:0/3692025930 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ac4004eb0 con 0x7f2ae00b17d0 2024-03-29T11:27:57.439 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.438+0000 7f2ad67fc640 10 client.?.objecter ms_handle_connect 0x7f2ae01422b0 2024-03-29T11:27:57.439 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.438+0000 7f2ad67fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:27:57.439 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.438+0000 7f2ad67fc640 1 -- 172.21.15.86:0/3692025930 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2828492702 0 0) 0x7f2ac4004eb0 con 0x7f2ae00b17d0 2024-03-29T11:27:57.439 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.439+0000 7f2ad67fc640 1 -- 172.21.15.86:0/3692025930 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ae00ab950 con 0x7f2ae00b17d0 2024-03-29T11:27:57.440 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.439+0000 7f2ad67fc640 1 -- 172.21.15.86:0/3692025930 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f2ad8001e80 con 0x7f2ae00b17d0 2024-03-29T11:27:57.440 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.439+0000 7f2ad67fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:57.440 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.439+0000 7f2ad67fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:27:57.440 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.439+0000 7f2ad67fc640 10 monclient(hunting): dump: 2024-03-29T11:27:57.440 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:57.440 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:57.440 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:57.440 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:57.440 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:57.440 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:57.440 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:57.440 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:57.440 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:57.440 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:57.440 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.439+0000 7f2ad67fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:27:57.441 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.439+0000 7f2ad67fc640 1 -- 172.21.15.86:0/3692025930 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2198985276 0 0) 0x7f2ae006faa0 con 0x7f2ae01422b0 2024-03-29T11:27:57.441 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.439+0000 7f2ad67fc640 10 monclient(hunting): _negotiate 2024-03-29T11:27:57.441 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.439+0000 7f2ad67fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:27:57.441 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.439+0000 7f2ad67fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:27:57.441 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.439+0000 7f2ad67fc640 10 monclient(hunting): my global_id is 4467 2024-03-29T11:27:57.441 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.439+0000 7f2ad67fc640 1 -- 172.21.15.86:0/3692025930 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2ac4004eb0 con 0x7f2ae01422b0 2024-03-29T11:27:57.441 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.439+0000 7f2ad67fc640 1 -- 172.21.15.86:0/3692025930 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2649500926 0 0) 0x7f2ac4004eb0 con 0x7f2ae01422b0 2024-03-29T11:27:57.441 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.439+0000 7f2ad67fc640 1 -- 172.21.15.86:0/3692025930 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2ae006faa0 con 0x7f2ae01422b0 2024-03-29T11:27:57.441 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.439+0000 7f2ad67fc640 1 -- 172.21.15.86:0/3692025930 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2371049289 0 0) 0x7f2ae00ab950 con 0x7f2ae00b17d0 2024-03-29T11:27:57.441 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.439+0000 7f2ad67fc640 1 -- 172.21.15.86:0/3692025930 >> v1:172.21.15.196:6789/0 conn(0x7f2ae0142690 legacy=0x7f2ae01dda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:57.441 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.440+0000 7f2ad67fc640 1 -- 172.21.15.86:0/3692025930 >> v1:172.21.15.86:6789/0 conn(0x7f2ae01422b0 legacy=0x7f2ae012ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:57.441 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.440+0000 7f2ad67fc640 10 monclient: _finish_hunting 0 2024-03-29T11:27:57.441 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.440+0000 7f2ad67fc640 1 monclient: found mon.c 2024-03-29T11:27:57.441 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.440+0000 7f2ad67fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:27:57.441 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.440+0000 7f2ad67fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:27:57.442 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.440+0000 7f2ad67fc640 1 -- 172.21.15.86:0/3692025930 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae006ff90 con 0x7f2ae00b17d0 2024-03-29T11:27:57.442 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.440+0000 7f2ad67fc640 10 monclient: _finish_auth 0 2024-03-29T11:27:57.442 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.440+0000 7f2ad67fc640 10 monclient: _check_auth_tickets 2024-03-29T11:27:57.442 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.440+0000 7f2ad67fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:27:57.442 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.440+0000 7f2ad67fc640 1 -- 172.21.15.86:0/3692025930 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f2ad8004e10 con 0x7f2ae00b17d0 2024-03-29T11:27:57.442 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.440+0000 7f2ad67fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:27:57.442 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.440+0000 7f2ad67fc640 1 -- 172.21.15.86:0/3692025930 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f2ad80051a0 con 0x7f2ae00b17d0 2024-03-29T11:27:57.442 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.440+0000 7f2ad67fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:27:57.442 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.440+0000 7f2ad67fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:27:57.442 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.440+0000 7f2ad67fc640 10 monclient: dump: 2024-03-29T11:27:57.442 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:27:57.442 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:27:57.442 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:57.443 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:27:57.443 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:27:57.443 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:27:57.443 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:27:57.443 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:27:57.443 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:27:57.443 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:27:57.443 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.440+0000 7f2ad7fff640 4 set_mon_vals no callback set 2024-03-29T11:27:57.443 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.440+0000 7f2ae7cb3640 5 monclient: authenticate success, global_id 4541 2024-03-29T11:27:57.443 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.440+0000 7f2ae7cb3640 10 monclient: _renew_subs 2024-03-29T11:27:57.443 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.440+0000 7f2ae7cb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:27:57.443 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.440+0000 7f2ae7cb3640 1 -- 172.21.15.86:0/3692025930 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ae012b570 con 0x7f2ae00b17d0 2024-03-29T11:27:57.443 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.440+0000 7f2ae7cb3640 10 client.4541.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:27:57.443 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.440+0000 7f2ae7cb3640 10 monclient: _renew_subs 2024-03-29T11:27:57.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.440+0000 7f2ae7cb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:27:57.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.440+0000 7f2ae7cb3640 1 -- 172.21.15.86:0/3692025930 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ae01df150 con 0x7f2ae00b17d0 2024-03-29T11:27:57.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.440+0000 7f2ae7cb3640 1 librados: init done 2024-03-29T11:27:57.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.441+0000 7f2ae7cb3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:57.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.442+0000 7f2ae7cb3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:57.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.442+0000 7f2ae7cb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:27:57.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.442+0000 7f2ae7cb3640 1 -- 172.21.15.86:0/3692025930 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ae0075f10 con 0x7f2ae00b17d0 2024-03-29T11:27:57.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.443+0000 7f2ad67fc640 1 -- 172.21.15.86:0/3692025930 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f2ad801b170 con 0x7f2ae00b17d0 2024-03-29T11:27:57.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.444+0000 7f2ad67fc640 1 -- 172.21.15.86:0/3692025930 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f2ad8084ec0 con 0x7f2ae00b17d0 2024-03-29T11:27:57.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.444+0000 7f2ae524e640 10 monclient: get_auth_request con 0x7f2ac406de40 auth_method 0 2024-03-29T11:27:57.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.444+0000 7f2ad67fc640 10 client.4541.objecter ms_dispatch 0x7f2ae0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:27:57.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.444+0000 7f2ad67fc640 3 client.4541.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:27:57.445 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.444+0000 7f2ad67fc640 3 client.4541.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:27:57.445 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.444+0000 7f2ad67fc640 20 client.4541.objecter dump_active .. 0 homeless 2024-03-29T11:27:57.445 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.444+0000 7f2ad67fc640 10 client.4541.objecter ms_handle_connect 0x7f2ac406de40 2024-03-29T11:27:57.445 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.451+0000 7f2ad67fc640 1 -- 172.21.15.86:0/3692025930 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f2ad80507a0 con 0x7f2ae00b17d0 2024-03-29T11:27:57.445 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.451+0000 7f2ad67fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:27:57.446 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.451+0000 7f2ad67fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:27:57.761 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.772+0000 7f2ae7cb3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:57.761 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.772+0000 7f2ae7cb3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:57.761 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.772+0000 7f2ae7cb3640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:27:57.761 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.772+0000 7f2ae7cb3640 1 -- 172.21.15.86:0/3692025930 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2ae0070120 con 0x7f2ae00b17d0 2024-03-29T11:27:57.763 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.774+0000 7f2ad67fc640 1 -- 172.21.15.86:0/3692025930 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f2ad80526d0 con 0x7f2ae00b17d0 2024-03-29T11:27:57.763 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.774+0000 7f2ad67fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:27:57.763 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.774+0000 7f2ad67fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:27:57.763 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:27:57.763 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:27:57.767 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.777+0000 7f2ae7cb3640 10 librados: watch_flush enter 2024-03-29T11:27:57.767 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.777+0000 7f2ae7cb3640 10 librados: watch_flush exit 2024-03-29T11:27:57.767 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.777+0000 7f2ae7cb3640 20 client.4541.objecter shutdown clearing up homeless session... 2024-03-29T11:27:57.767 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.777+0000 7f2ae7cb3640 10 client.4541.objecter successfully canceled tick 2024-03-29T11:27:57.767 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.778+0000 7f2ae7cb3640 1 -- 172.21.15.86:0/3692025930 >> v1:172.21.15.196:6816/588110318 conn(0x7f2ac406de40 legacy=0x7f2ac4070300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:57.767 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.778+0000 7f2ae7cb3640 10 monclient: shutdown 2024-03-29T11:27:57.767 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.778+0000 7f2ae7cb3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:27:57.767 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.778+0000 7f2ae7cb3640 1 -- 172.21.15.86:0/3692025930 >> v1:172.21.15.86:6790/0 conn(0x7f2ae00b17d0 legacy=0x7f2ae0070d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:27:57.767 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.778+0000 7f2ae7cb3640 1 -- 172.21.15.86:0/3692025930 shutdown_connections 2024-03-29T11:27:57.768 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.778+0000 7f2ae7cb3640 1 -- 172.21.15.86:0/3692025930 >> 172.21.15.86:0/3692025930 conn(0x7f2ae0066a00 msgr2=0x7f2ae013dbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:27:57.768 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.778+0000 7f2ae7cb3640 1 -- 172.21.15.86:0/3692025930 shutdown_connections 2024-03-29T11:27:57.768 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.778+0000 7f2ae7cb3640 1 -- 172.21.15.86:0/3692025930 wait complete. 2024-03-29T11:27:57.768 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:27:57.778+0000 7f2ae7cb3640 1 librados: shutdown 2024-03-29T11:27:57.782 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:04.590 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:04.751 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.762+0000 7fc1a3e9d640 10 monclient: get_monmap_and_config 2024-03-29T11:28:04.751 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.762+0000 7fc1a3e9d640 10 monclient: build_initial_monmap 2024-03-29T11:28:04.751 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.762+0000 7fc1a3e9d640 10 monclient: monmap: 2024-03-29T11:28:04.751 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:28:04.752 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:04.752 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:28:04.763453+0000 2024-03-29T11:28:04.752 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:28:04.763453+0000 2024-03-29T11:28:04.752 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:04.752 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:04.752 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:28:04.752 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:28:04.752 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:28:04.752 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:04.752 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.763+0000 7fc1a3e9d640 1 Processor -- start 2024-03-29T11:28:04.753 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.763+0000 7fc1a3e9d640 1 -- start start 2024-03-29T11:28:04.753 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.763+0000 7fc1a3e9d640 10 monclient: init 2024-03-29T11:28:04.754 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.764+0000 7fc1a3e9d640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:04.754 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.764+0000 7fc1a3e9d640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:28:04.754 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.764+0000 7fc1a3e9d640 10 monclient(hunting): picked mon.noname-c con 0x7fc19c06c1e0 addr v1:172.21.15.196:6789/0 2024-03-29T11:28:04.754 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.764+0000 7fc1a3e9d640 10 monclient(hunting): picked mon.noname-a con 0x7fc19c06c5c0 addr v1:172.21.15.86:6789/0 2024-03-29T11:28:04.754 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.764+0000 7fc1a3e9d640 10 monclient(hunting): picked mon.noname-b con 0x7fc19c0ae590 addr v1:172.21.15.86:6790/0 2024-03-29T11:28:04.754 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.764+0000 7fc1a3e9d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc19c0ade90 con 0x7fc19c06c5c0 2024-03-29T11:28:04.754 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.764+0000 7fc1a3e9d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc19c0ae170 con 0x7fc19c06c1e0 2024-03-29T11:28:04.754 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.764+0000 7fc1a3e9d640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc19c1422a0 con 0x7fc19c0ae590 2024-03-29T11:28:04.754 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.764+0000 7fc1a3e9d640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:04.754 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.764+0000 7fc1a3e9d640 10 monclient(hunting): authenticate will time out at 1309.082786s 2024-03-29T11:28:04.755 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.764+0000 7fc1a1c39640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fc19c06c1e0 0x7fc19c1427c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:33070/0 (socket says 172.21.15.86:33070) 2024-03-29T11:28:04.755 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.764+0000 7fc1a1c39640 1 -- 172.21.15.86:0/4293350291 learned_addr learned my addr 172.21.15.86:0/4293350291 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:28:04.755 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.765+0000 7fc1a0c37640 1 -- 172.21.15.86:0/4293350291 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 97593526 0 0) 0x7fc19c0ae170 con 0x7fc19c06c1e0 2024-03-29T11:28:04.755 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.765+0000 7fc1a0c37640 10 monclient(hunting): _negotiate 2024-03-29T11:28:04.755 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.765+0000 7fc1a0c37640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:04.755 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.765+0000 7fc1a0c37640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:04.755 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.765+0000 7fc1a0c37640 10 monclient(hunting): my global_id is 4453 2024-03-29T11:28:04.755 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.765+0000 7fc1a0c37640 1 -- 172.21.15.86:0/4293350291 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc188003520 con 0x7fc19c06c1e0 2024-03-29T11:28:04.755 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.765+0000 7fc1a0c37640 1 -- 172.21.15.86:0/4293350291 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1972919763 0 0) 0x7fc19c0ade90 con 0x7fc19c06c5c0 2024-03-29T11:28:04.756 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.765+0000 7fc1a0c37640 10 monclient(hunting): _negotiate 2024-03-29T11:28:04.756 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.765+0000 7fc1a0c37640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:04.756 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.765+0000 7fc1a0c37640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:04.756 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.765+0000 7fc1a0c37640 10 monclient(hunting): my global_id is 4470 2024-03-29T11:28:04.756 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.765+0000 7fc1a0c37640 1 -- 172.21.15.86:0/4293350291 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc19c0ae170 con 0x7fc19c06c5c0 2024-03-29T11:28:04.756 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.765+0000 7fc1a0c37640 1 -- 172.21.15.86:0/4293350291 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 378880408 0 0) 0x7fc188003520 con 0x7fc19c06c1e0 2024-03-29T11:28:04.756 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.766+0000 7fc1a0c37640 1 -- 172.21.15.86:0/4293350291 >> v1:172.21.15.86:6790/0 conn(0x7fc19c0ae590 legacy=0x7fc19c14d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:04.756 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.766+0000 7fc1a0c37640 1 -- 172.21.15.86:0/4293350291 >> v1:172.21.15.86:6789/0 conn(0x7fc19c06c5c0 legacy=0x7fc19c13f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:04.756 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.766+0000 7fc1a0c37640 10 monclient: _finish_hunting 0 2024-03-29T11:28:04.756 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.766+0000 7fc1a0c37640 1 monclient: found mon.noname-c 2024-03-29T11:28:04.756 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.766+0000 7fc1a0c37640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:04.756 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.766+0000 7fc1a0c37640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T11:28:04.756 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.766+0000 7fc1a0c37640 1 -- 172.21.15.86:0/4293350291 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc19c142ed0 con 0x7fc19c06c1e0 2024-03-29T11:28:04.756 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.766+0000 7fc1a0c37640 10 monclient: _finish_auth 0 2024-03-29T11:28:04.757 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.766+0000 7fc1a0c37640 10 monclient: _check_auth_tickets 2024-03-29T11:28:04.757 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.766+0000 7fc1a0c37640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:04.757 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.766+0000 7fc1a0c37640 1 -- 172.21.15.86:0/4293350291 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc198001a10 con 0x7fc19c06c1e0 2024-03-29T11:28:04.757 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.766+0000 7fc1a0c37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:04.757 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.766+0000 7fc1a0c37640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:04.757 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.766+0000 7fc1a0c37640 10 monclient: dump: 2024-03-29T11:28:04.757 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:28:04.757 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:04.757 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:04.757 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:04.757 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:04.757 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:04.757 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:28:04.757 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:28:04.757 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:28:04.758 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:04.758 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.766+0000 7fc1a3e9d640 5 monclient: authenticate success, global_id 4453 2024-03-29T11:28:04.758 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.766+0000 7fc1a3e9d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:04.758 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.766+0000 7fc1a0c37640 1 -- 172.21.15.86:0/4293350291 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc198001b70 con 0x7fc19c06c1e0 2024-03-29T11:28:04.758 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.766+0000 7fc1a0c37640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:04.758 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.766+0000 7fc1a0c37640 1 -- 172.21.15.86:0/4293350291 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc198001f80 con 0x7fc19c06c1e0 2024-03-29T11:28:04.758 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.766+0000 7fc1a0c37640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:04.758 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.766+0000 7fc1a0c37640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:04.758 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.766+0000 7fc1a0c37640 10 monclient: dump: 2024-03-29T11:28:04.758 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:28:04.758 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:04.758 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:04.758 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:04.758 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:04.758 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:04.759 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:28:04.759 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:28:04.759 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:28:04.759 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:04.759 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.766+0000 7fc1a3e9d640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:04.759 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.766+0000 7fc1a3e9d640 4 set_mon_vals no callback set 2024-03-29T11:28:04.765 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.767+0000 7fc1a3e9d640 10 monclient: shutdown 2024-03-29T11:28:04.765 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.767+0000 7fc1a3e9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:04.765 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.767+0000 7fc1a3e9d640 1 -- 172.21.15.86:0/4293350291 >> v1:172.21.15.196:6789/0 conn(0x7fc19c06c1e0 legacy=0x7fc19c1427c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:04.765 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.767+0000 7fc1a3e9d640 1 -- 172.21.15.86:0/4293350291 shutdown_connections 2024-03-29T11:28:04.766 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.767+0000 7fc1a3e9d640 1 -- 172.21.15.86:0/4293350291 >> 172.21.15.86:0/4293350291 conn(0x7fc19c067500 msgr2=0x7fc19c0b5a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:04.766 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.768+0000 7fc1a3e9d640 1 -- 172.21.15.86:0/4293350291 shutdown_connections 2024-03-29T11:28:04.766 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.768+0000 7fc1a3e9d640 1 -- 172.21.15.86:0/4293350291 wait complete. 2024-03-29T11:28:04.766 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.769+0000 7fc1a3e9d640 10 monclient: build_initial_monmap 2024-03-29T11:28:04.766 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.769+0000 7fc1a3e9d640 10 monclient: monmap: 2024-03-29T11:28:04.766 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:28:04.766 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:04.766 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:28:04.766 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:28:04.766 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:04.766 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:04.766 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:28:04.766 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:28:04.767 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:28:04.767 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:04.767 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.770+0000 7fc1a3e9d640 1 librados: starting msgr at 2024-03-29T11:28:04.767 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.770+0000 7fc1a3e9d640 1 librados: starting objecter 2024-03-29T11:28:04.767 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.770+0000 7fc1a3e9d640 1 Processor -- start 2024-03-29T11:28:04.767 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.770+0000 7fc1a3e9d640 1 -- start start 2024-03-29T11:28:04.767 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.770+0000 7fc1a3e9d640 1 librados: setting wanted keys 2024-03-29T11:28:04.767 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.770+0000 7fc1a3e9d640 1 librados: calling monclient init 2024-03-29T11:28:04.767 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.770+0000 7fc1a3e9d640 10 monclient: init 2024-03-29T11:28:04.767 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.771+0000 7fc1a3e9d640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:04.767 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.771+0000 7fc1a3e9d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:04.767 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.771+0000 7fc1a3e9d640 10 monclient(hunting): picked mon.noname-b con 0x7fc19c06c1e0 addr v1:172.21.15.196:6789/0 2024-03-29T11:28:04.767 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.771+0000 7fc1a3e9d640 10 monclient(hunting): picked mon.noname-c con 0x7fc19c06c5c0 addr v1:172.21.15.86:6790/0 2024-03-29T11:28:04.767 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.771+0000 7fc1a3e9d640 10 monclient(hunting): picked mon.noname-a con 0x7fc19c0ae590 addr v1:172.21.15.86:6789/0 2024-03-29T11:28:04.767 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.771+0000 7fc1a3e9d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc19c072470 con 0x7fc19c0ae590 2024-03-29T11:28:04.768 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.771+0000 7fc1a3e9d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc19c1db710 con 0x7fc19c06c1e0 2024-03-29T11:28:04.768 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.771+0000 7fc1a3e9d640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc19c0700f0 con 0x7fc19c06c5c0 2024-03-29T11:28:04.768 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.771+0000 7fc1a3e9d640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:04.768 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.771+0000 7fc1a3e9d640 10 monclient(hunting): authenticate will time out at 1309.089431s 2024-03-29T11:28:04.768 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.771+0000 7fc1a243a640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fc19c0ae590 0x7fc19c070bb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:47804/0 (socket says 172.21.15.86:47804) 2024-03-29T11:28:04.768 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.771+0000 7fc1a243a640 1 -- 172.21.15.86:0/2166334446 learned_addr learned my addr 172.21.15.86:0/2166334446 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:28:04.768 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.771+0000 7fc1927fc640 10 client.?.objecter ms_handle_connect 0x7fc19c0ae590 2024-03-29T11:28:04.768 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.771+0000 7fc1927fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:04.768 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.772+0000 7fc1927fc640 10 client.?.objecter ms_handle_connect 0x7fc19c06c1e0 2024-03-29T11:28:04.768 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.772+0000 7fc1927fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:04.768 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.772+0000 7fc1927fc640 1 -- 172.21.15.86:0/2166334446 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2440881241 0 0) 0x7fc19c1db710 con 0x7fc19c06c1e0 2024-03-29T11:28:04.768 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.772+0000 7fc1927fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:04.768 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.772+0000 7fc1927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:04.768 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.772+0000 7fc1927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:04.769 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.772+0000 7fc1927fc640 10 monclient(hunting): my global_id is 4456 2024-03-29T11:28:04.769 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.772+0000 7fc1927fc640 1 -- 172.21.15.86:0/2166334446 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc180004eb0 con 0x7fc19c06c1e0 2024-03-29T11:28:04.769 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.772+0000 7fc1927fc640 10 client.?.objecter ms_handle_connect 0x7fc19c06c5c0 2024-03-29T11:28:04.769 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.772+0000 7fc1927fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:04.769 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.772+0000 7fc1927fc640 1 -- 172.21.15.86:0/2166334446 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2890861370 0 0) 0x7fc19c072470 con 0x7fc19c0ae590 2024-03-29T11:28:04.769 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.772+0000 7fc1927fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:04.769 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.773+0000 7fc1927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:04.769 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.773+0000 7fc1927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:04.769 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.773+0000 7fc1927fc640 10 monclient(hunting): my global_id is 4473 2024-03-29T11:28:04.769 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.773+0000 7fc1927fc640 1 -- 172.21.15.86:0/2166334446 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc19c1db710 con 0x7fc19c0ae590 2024-03-29T11:28:04.769 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.773+0000 7fc1927fc640 1 -- 172.21.15.86:0/2166334446 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 22387072 0 0) 0x7fc19c0700f0 con 0x7fc19c06c5c0 2024-03-29T11:28:04.769 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.773+0000 7fc1927fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:04.769 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.773+0000 7fc1927fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:04.769 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.773+0000 7fc1927fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:04.769 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.773+0000 7fc1927fc640 10 monclient(hunting): my global_id is 4547 2024-03-29T11:28:04.770 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.773+0000 7fc1927fc640 1 -- 172.21.15.86:0/2166334446 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc19c072470 con 0x7fc19c06c5c0 2024-03-29T11:28:04.770 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.773+0000 7fc1927fc640 1 -- 172.21.15.86:0/2166334446 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3776308912 0 0) 0x7fc180004eb0 con 0x7fc19c06c1e0 2024-03-29T11:28:04.770 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.773+0000 7fc1927fc640 1 -- 172.21.15.86:0/2166334446 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc19c0700f0 con 0x7fc19c06c1e0 2024-03-29T11:28:04.770 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.773+0000 7fc1927fc640 1 -- 172.21.15.86:0/2166334446 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc198001e80 con 0x7fc19c06c1e0 2024-03-29T11:28:04.770 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.773+0000 7fc1927fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:04.770 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.773+0000 7fc1927fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:04.770 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.773+0000 7fc1927fc640 10 monclient(hunting): dump: 2024-03-29T11:28:04.770 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:28:04.770 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:04.770 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:04.770 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:04.770 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:04.770 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:04.770 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:28:04.770 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:28:04.771 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:28:04.771 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:04.771 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.773+0000 7fc1927fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:04.771 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.774+0000 7fc1927fc640 1 -- 172.21.15.86:0/2166334446 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4066853007 0 0) 0x7fc19c0700f0 con 0x7fc19c06c1e0 2024-03-29T11:28:04.771 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.774+0000 7fc1927fc640 1 -- 172.21.15.86:0/2166334446 >> v1:172.21.15.86:6790/0 conn(0x7fc19c06c5c0 legacy=0x7fc19c1369b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:04.771 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.774+0000 7fc1927fc640 1 -- 172.21.15.86:0/2166334446 >> v1:172.21.15.86:6789/0 conn(0x7fc19c0ae590 legacy=0x7fc19c070bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:04.771 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.774+0000 7fc1927fc640 10 monclient: _finish_hunting 0 2024-03-29T11:28:04.771 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.774+0000 7fc1927fc640 1 monclient: found mon.b 2024-03-29T11:28:04.771 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.774+0000 7fc1927fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:04.771 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.774+0000 7fc1927fc640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:28:04.771 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.774+0000 7fc1927fc640 1 -- 172.21.15.86:0/2166334446 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc19c0b3a80 con 0x7fc19c06c1e0 2024-03-29T11:28:04.771 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.774+0000 7fc1927fc640 10 monclient: _finish_auth 0 2024-03-29T11:28:04.771 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.774+0000 7fc1927fc640 10 monclient: _check_auth_tickets 2024-03-29T11:28:04.772 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.774+0000 7fc1927fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:04.772 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.774+0000 7fc1a3e9d640 5 monclient: authenticate success, global_id 4456 2024-03-29T11:28:04.772 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.774+0000 7fc1a3e9d640 10 monclient: _renew_subs 2024-03-29T11:28:04.772 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.774+0000 7fc1a3e9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:28:04.772 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.774+0000 7fc1a3e9d640 1 -- 172.21.15.86:0/2166334446 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc19c1dc340 con 0x7fc19c06c1e0 2024-03-29T11:28:04.772 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.774+0000 7fc1a3e9d640 10 client.4456.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:04.772 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.774+0000 7fc1a3e9d640 10 monclient: _renew_subs 2024-03-29T11:28:04.772 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.774+0000 7fc1a3e9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:28:04.772 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.774+0000 7fc1a3e9d640 1 -- 172.21.15.86:0/2166334446 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc19c1372a0 con 0x7fc19c06c1e0 2024-03-29T11:28:04.772 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.774+0000 7fc1a3e9d640 1 librados: init done 2024-03-29T11:28:04.772 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.775+0000 7fc1927fc640 1 -- 172.21.15.86:0/2166334446 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc198001fe0 con 0x7fc19c06c1e0 2024-03-29T11:28:04.772 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.775+0000 7fc1927fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:04.772 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.775+0000 7fc193fff640 4 set_mon_vals no callback set 2024-03-29T11:28:04.774 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.776+0000 7fc1927fc640 1 -- 172.21.15.86:0/2166334446 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc198004fb0 con 0x7fc19c06c1e0 2024-03-29T11:28:04.774 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.776+0000 7fc1927fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:04.774 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.776+0000 7fc1927fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:04.775 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.776+0000 7fc1927fc640 10 monclient: dump: 2024-03-29T11:28:04.775 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:28:04.775 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:04.775 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:04.775 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:04.775 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:04.775 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:04.775 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:28:04.775 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:28:04.775 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:28:04.776 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:04.776 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.776+0000 7fc1a3e9d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:04.776 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.776+0000 7fc1a3e9d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:04.776 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.776+0000 7fc1a3e9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:28:04.776 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.776+0000 7fc1a3e9d640 1 -- 172.21.15.86:0/2166334446 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc184005080 con 0x7fc19c06c1e0 2024-03-29T11:28:04.776 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.778+0000 7fc1927fc640 1 -- 172.21.15.86:0/2166334446 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7fc198005110 con 0x7fc19c06c1e0 2024-03-29T11:28:04.776 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.778+0000 7fc1927fc640 1 -- 172.21.15.86:0/2166334446 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7fc198084e40 con 0x7fc19c06c1e0 2024-03-29T11:28:04.776 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.779+0000 7fc1a1438640 10 monclient: get_auth_request con 0x7fc18006de80 auth_method 0 2024-03-29T11:28:04.776 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.778+0000 7fc1927fc640 10 client.4456.objecter ms_dispatch 0x7fc19c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:04.776 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.779+0000 7fc1927fc640 3 client.4456.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:04.776 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.779+0000 7fc1927fc640 3 client.4456.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:04.776 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.779+0000 7fc1927fc640 20 client.4456.objecter dump_active .. 0 homeless 2024-03-29T11:28:04.776 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.779+0000 7fc1927fc640 10 client.4456.objecter ms_handle_connect 0x7fc18006de80 2024-03-29T11:28:04.776 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.784+0000 7fc1927fc640 1 -- 172.21.15.86:0/2166334446 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc198050720 con 0x7fc19c06c1e0 2024-03-29T11:28:04.776 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.784+0000 7fc1927fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:04.777 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:04.784+0000 7fc1927fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:05.228 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:05.239+0000 7fc1a3e9d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:05.228 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:05.239+0000 7fc1a3e9d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:05.229 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:05.239+0000 7fc1a3e9d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:28:05.229 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:05.239+0000 7fc1a3e9d640 1 -- 172.21.15.86:0/2166334446 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc184005330 con 0x7fc19c06c1e0 2024-03-29T11:28:05.229 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:05.240+0000 7fc1927fc640 1 -- 172.21.15.86:0/2166334446 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7fc198052650 con 0x7fc19c06c1e0 2024-03-29T11:28:05.229 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:05.240+0000 7fc1927fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:05.229 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:05.240+0000 7fc1927fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:05.230 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:28:05.230 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:05.233 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:05.244+0000 7fc1a3e9d640 10 librados: watch_flush enter 2024-03-29T11:28:05.233 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:05.244+0000 7fc1a3e9d640 10 librados: watch_flush exit 2024-03-29T11:28:05.233 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:05.244+0000 7fc1a3e9d640 20 client.4456.objecter shutdown clearing up homeless session... 2024-03-29T11:28:05.233 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:05.244+0000 7fc1a3e9d640 10 client.4456.objecter successfully canceled tick 2024-03-29T11:28:05.234 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:05.244+0000 7fc1a3e9d640 1 -- 172.21.15.86:0/2166334446 >> v1:172.21.15.196:6816/588110318 conn(0x7fc18006de80 legacy=0x7fc180070340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:05.234 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:05.244+0000 7fc1a3e9d640 10 monclient: shutdown 2024-03-29T11:28:05.234 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:05.244+0000 7fc1a3e9d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:05.234 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:05.244+0000 7fc1a3e9d640 1 -- 172.21.15.86:0/2166334446 >> v1:172.21.15.196:6789/0 conn(0x7fc19c06c1e0 legacy=0x7fc19c1dbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:05.234 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:05.244+0000 7fc1a3e9d640 1 -- 172.21.15.86:0/2166334446 shutdown_connections 2024-03-29T11:28:05.234 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:05.244+0000 7fc1a3e9d640 1 -- 172.21.15.86:0/2166334446 >> 172.21.15.86:0/2166334446 conn(0x7fc19c067500 msgr2=0x7fc19c069b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:05.234 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:05.244+0000 7fc1a3e9d640 1 -- 172.21.15.86:0/2166334446 shutdown_connections 2024-03-29T11:28:05.234 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:05.244+0000 7fc1a3e9d640 1 -- 172.21.15.86:0/2166334446 wait complete. 2024-03-29T11:28:05.234 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:05.244+0000 7fc1a3e9d640 1 librados: shutdown 2024-03-29T11:28:05.249 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:12.055 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:12.209 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.218+0000 7feb1619d640 10 monclient: get_monmap_and_config 2024-03-29T11:28:12.209 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.218+0000 7feb1619d640 10 monclient: build_initial_monmap 2024-03-29T11:28:12.209 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.218+0000 7feb1619d640 10 monclient: monmap: 2024-03-29T11:28:12.209 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:28:12.209 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:12.209 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:28:12.219501+0000 2024-03-29T11:28:12.209 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:28:12.219501+0000 2024-03-29T11:28:12.209 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:12.209 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:12.209 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:28:12.209 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:28:12.209 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:28:12.209 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:12.213 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.223+0000 7feb1619d640 1 Processor -- start 2024-03-29T11:28:12.213 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.224+0000 7feb1619d640 1 -- start start 2024-03-29T11:28:12.213 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.224+0000 7feb1619d640 10 monclient: init 2024-03-29T11:28:12.215 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.225+0000 7feb1619d640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:12.215 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.225+0000 7feb1619d640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:28:12.215 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.225+0000 7feb1619d640 10 monclient(hunting): picked mon.noname-b con 0x7feb100aa2a0 addr v1:172.21.15.86:6790/0 2024-03-29T11:28:12.215 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.225+0000 7feb1619d640 10 monclient(hunting): picked mon.noname-c con 0x7feb10141f20 addr v1:172.21.15.196:6789/0 2024-03-29T11:28:12.215 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.225+0000 7feb1619d640 10 monclient(hunting): picked mon.noname-a con 0x7feb1013b3c0 addr v1:172.21.15.86:6789/0 2024-03-29T11:28:12.215 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.225+0000 7feb1619d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb1013e5c0 con 0x7feb1013b3c0 2024-03-29T11:28:12.215 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.225+0000 7feb1619d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb100ab190 con 0x7feb10141f20 2024-03-29T11:28:12.215 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.225+0000 7feb1619d640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb100ab470 con 0x7feb100aa2a0 2024-03-29T11:28:12.215 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.225+0000 7feb1619d640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:12.215 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.225+0000 7feb1619d640 10 monclient(hunting): authenticate will time out at 1316.543277s 2024-03-29T11:28:12.217 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.226+0000 7feb0ffff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7feb1013b3c0 0x7feb1013b810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:46820/0 (socket says 172.21.15.86:46820) 2024-03-29T11:28:12.218 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.226+0000 7feb0ffff640 1 -- 172.21.15.86:0/375234818 learned_addr learned my addr 172.21.15.86:0/375234818 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:28:12.218 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.226+0000 7feb0e7fc640 1 -- 172.21.15.86:0/375234818 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1511614960 0 0) 0x7feb1013e5c0 con 0x7feb1013b3c0 2024-03-29T11:28:12.218 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.226+0000 7feb0e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:12.218 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.226+0000 7feb0e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:12.218 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.226+0000 7feb0e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:12.218 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.226+0000 7feb0e7fc640 10 monclient(hunting): my global_id is 4476 2024-03-29T11:28:12.218 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.226+0000 7feb0e7fc640 1 -- 172.21.15.86:0/375234818 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaf8003440 con 0x7feb1013b3c0 2024-03-29T11:28:12.218 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.227+0000 7feb0e7fc640 1 -- 172.21.15.86:0/375234818 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3366156639 0 0) 0x7feaf8003440 con 0x7feb1013b3c0 2024-03-29T11:28:12.218 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.227+0000 7feb0e7fc640 1 -- 172.21.15.86:0/375234818 >> v1:172.21.15.86:6790/0 conn(0x7feb100aa2a0 legacy=0x7feb100ad270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:12.222 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.228+0000 7feb0e7fc640 1 -- 172.21.15.86:0/375234818 >> v1:172.21.15.196:6789/0 conn(0x7feb10141f20 legacy=0x7feb10142340 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:28:12.222 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.228+0000 7feb0e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:28:12.222 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.228+0000 7feb0e7fc640 1 monclient: found mon.noname-a 2024-03-29T11:28:12.222 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.228+0000 7feb0e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:12.222 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.228+0000 7feb0e7fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.86:6789/0 2024-03-29T11:28:12.223 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.228+0000 7feb0e7fc640 1 -- 172.21.15.86:0/375234818 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb10142a50 con 0x7feb1013b3c0 2024-03-29T11:28:12.223 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.228+0000 7feb0e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:28:12.223 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.228+0000 7feb0e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:28:12.223 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.228+0000 7feb0e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:12.223 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.228+0000 7feb0e7fc640 1 -- 172.21.15.86:0/375234818 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7feb04001690 con 0x7feb1013b3c0 2024-03-29T11:28:12.223 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.228+0000 7feb0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:12.223 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.228+0000 7feb0e7fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:12.223 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.228+0000 7feb0e7fc640 10 monclient: dump: 2024-03-29T11:28:12.223 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:28:12.223 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:12.223 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:12.223 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:12.223 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:12.223 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:12.223 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:28:12.224 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:28:12.224 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:28:12.224 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:12.224 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.228+0000 7feb1619d640 5 monclient: authenticate success, global_id 4476 2024-03-29T11:28:12.224 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.228+0000 7feb1619d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:12.224 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.229+0000 7feb0e7fc640 1 -- 172.21.15.86:0/375234818 <== mon.0 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb040019a0 con 0x7feb1013b3c0 2024-03-29T11:28:12.224 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.229+0000 7feb0e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:12.224 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.229+0000 7feb1619d640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:12.224 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.229+0000 7feb1619d640 4 set_mon_vals no callback set 2024-03-29T11:28:12.224 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.229+0000 7feb0e7fc640 1 -- 172.21.15.86:0/375234818 <== mon.0 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7feb04001d30 con 0x7feb1013b3c0 2024-03-29T11:28:12.224 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.229+0000 7feb0e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:12.224 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.229+0000 7feb0e7fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:12.224 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.229+0000 7feb0e7fc640 10 monclient: dump: 2024-03-29T11:28:12.224 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:28:12.224 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:12.225 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:12.225 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:12.225 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:12.225 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:12.225 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:28:12.225 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:28:12.225 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:28:12.225 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:12.225 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.229+0000 7feb1619d640 10 monclient: shutdown 2024-03-29T11:28:12.225 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.229+0000 7feb1619d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:12.225 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.229+0000 7feb1619d640 1 -- 172.21.15.86:0/375234818 >> v1:172.21.15.86:6789/0 conn(0x7feb1013b3c0 legacy=0x7feb1013b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:12.225 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.230+0000 7feb1619d640 1 -- 172.21.15.86:0/375234818 shutdown_connections 2024-03-29T11:28:12.225 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.230+0000 7feb1619d640 1 -- 172.21.15.86:0/375234818 >> 172.21.15.86:0/375234818 conn(0x7feb100b35e0 msgr2=0x7feb100b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:12.225 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.230+0000 7feb1619d640 1 -- 172.21.15.86:0/375234818 shutdown_connections 2024-03-29T11:28:12.226 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.230+0000 7feb1619d640 1 -- 172.21.15.86:0/375234818 wait complete. 2024-03-29T11:28:12.226 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.232+0000 7feb1619d640 10 monclient: build_initial_monmap 2024-03-29T11:28:12.226 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.232+0000 7feb1619d640 10 monclient: monmap: 2024-03-29T11:28:12.226 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:28:12.226 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:12.226 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:28:12.226 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:28:12.226 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:12.226 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:12.226 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:28:12.226 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:28:12.226 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:28:12.226 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:12.226 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.232+0000 7feb1619d640 1 librados: starting msgr at 2024-03-29T11:28:12.226 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.232+0000 7feb1619d640 1 librados: starting objecter 2024-03-29T11:28:12.227 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.232+0000 7feb1619d640 1 Processor -- start 2024-03-29T11:28:12.227 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.233+0000 7feb1619d640 1 -- start start 2024-03-29T11:28:12.227 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.233+0000 7feb1619d640 1 librados: setting wanted keys 2024-03-29T11:28:12.227 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.233+0000 7feb1619d640 1 librados: calling monclient init 2024-03-29T11:28:12.227 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.233+0000 7feb1619d640 10 monclient: init 2024-03-29T11:28:12.229 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.234+0000 7feb1619d640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:12.229 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.234+0000 7feb1619d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:12.229 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.235+0000 7feb1619d640 10 monclient(hunting): picked mon.noname-b con 0x7feb100aa2a0 addr v1:172.21.15.196:6789/0 2024-03-29T11:28:12.229 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.235+0000 7feb1619d640 10 monclient(hunting): picked mon.noname-c con 0x7feb1013b3c0 addr v1:172.21.15.86:6790/0 2024-03-29T11:28:12.229 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.235+0000 7feb1619d640 10 monclient(hunting): picked mon.noname-a con 0x7feb10141f20 addr v1:172.21.15.86:6789/0 2024-03-29T11:28:12.229 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.235+0000 7feb1619d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb10120960 con 0x7feb10141f20 2024-03-29T11:28:12.229 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.235+0000 7feb1619d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb10072270 con 0x7feb100aa2a0 2024-03-29T11:28:12.230 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.235+0000 7feb1619d640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb10071550 con 0x7feb1013b3c0 2024-03-29T11:28:12.230 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.235+0000 7feb1619d640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:12.230 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.235+0000 7feb1619d640 10 monclient(hunting): authenticate will time out at 1316.553809s 2024-03-29T11:28:12.230 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.236+0000 7feb0ffff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7feb10141f20 0x7feb101e2a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:49388/0 (socket says 172.21.15.86:49388) 2024-03-29T11:28:12.230 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.236+0000 7feb0ffff640 1 -- 172.21.15.86:0/2190546550 learned_addr learned my addr 172.21.15.86:0/2190546550 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:28:12.230 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.236+0000 7feaf7fff640 10 client.?.objecter ms_handle_connect 0x7feb10141f20 2024-03-29T11:28:12.230 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.236+0000 7feaf7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:12.230 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.236+0000 7feaf7fff640 1 -- 172.21.15.86:0/2190546550 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3999459131 0 0) 0x7feb10120960 con 0x7feb10141f20 2024-03-29T11:28:12.230 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.236+0000 7feaf7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:12.230 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.236+0000 7feaf7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:12.230 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.236+0000 7feaf7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:12.230 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.236+0000 7feaf7fff640 10 monclient(hunting): my global_id is 4479 2024-03-29T11:28:12.230 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.236+0000 7feaf7fff640 1 -- 172.21.15.86:0/2190546550 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feafc004eb0 con 0x7feb10141f20 2024-03-29T11:28:12.231 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.237+0000 7feaf7fff640 1 -- 172.21.15.86:0/2190546550 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3812236316 0 0) 0x7feafc004eb0 con 0x7feb10141f20 2024-03-29T11:28:12.231 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.237+0000 7feaf7fff640 1 -- 172.21.15.86:0/2190546550 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb10120960 con 0x7feb10141f20 2024-03-29T11:28:12.231 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.237+0000 7feaf7fff640 1 -- 172.21.15.86:0/2190546550 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7feb04001650 con 0x7feb10141f20 2024-03-29T11:28:12.231 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.237+0000 7feaf7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:12.231 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.237+0000 7feaf7fff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:12.231 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.237+0000 7feaf7fff640 10 monclient(hunting): dump: 2024-03-29T11:28:12.231 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:28:12.231 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:12.231 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:12.231 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:12.231 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:12.231 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:12.231 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:28:12.231 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:28:12.231 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:28:12.232 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:12.232 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.237+0000 7feaf7fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:12.232 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.237+0000 7feaf7fff640 10 client.?.objecter ms_handle_connect 0x7feb100aa2a0 2024-03-29T11:28:12.232 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.237+0000 7feaf7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:12.232 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.237+0000 7feaf7fff640 1 -- 172.21.15.86:0/2190546550 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1158563385 0 0) 0x7feb10072270 con 0x7feb100aa2a0 2024-03-29T11:28:12.232 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.237+0000 7feaf7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:12.232 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.237+0000 7feaf7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:12.232 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.237+0000 7feaf7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:12.232 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.237+0000 7feaf7fff640 10 monclient(hunting): my global_id is 4459 2024-03-29T11:28:12.232 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.237+0000 7feaf7fff640 1 -- 172.21.15.86:0/2190546550 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feafc004eb0 con 0x7feb100aa2a0 2024-03-29T11:28:12.232 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.238+0000 7feaf7fff640 1 -- 172.21.15.86:0/2190546550 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1701574318 0 0) 0x7feafc004eb0 con 0x7feb100aa2a0 2024-03-29T11:28:12.232 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.238+0000 7feaf7fff640 1 -- 172.21.15.86:0/2190546550 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7feb10072270 con 0x7feb100aa2a0 2024-03-29T11:28:12.232 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.238+0000 7feaf7fff640 1 -- 172.21.15.86:0/2190546550 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7feb000016f0 con 0x7feb100aa2a0 2024-03-29T11:28:12.232 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.238+0000 7feaf7fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:12.232 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.238+0000 7feaf7fff640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:12.232 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.238+0000 7feaf7fff640 10 monclient(hunting): dump: 2024-03-29T11:28:12.233 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:28:12.233 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:12.233 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:12.233 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:12.233 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:12.233 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:12.233 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:28:12.233 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:28:12.233 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:28:12.233 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:12.233 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.238+0000 7feaf7fff640 1 -- 172.21.15.86:0/2190546550 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 494322999 0 0) 0x7feb10072270 con 0x7feb100aa2a0 2024-03-29T11:28:12.233 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.238+0000 7feaf7fff640 1 -- 172.21.15.86:0/2190546550 >> v1:172.21.15.86:6790/0 conn(0x7feb1013b3c0 legacy=0x7feb101da940 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:12.234 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.242+0000 7feaf7fff640 1 -- 172.21.15.86:0/2190546550 >> v1:172.21.15.86:6789/0 conn(0x7feb10141f20 legacy=0x7feb101e2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:12.235 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.242+0000 7feaf7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:28:12.235 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.242+0000 7feaf7fff640 1 monclient: found mon.b 2024-03-29T11:28:12.235 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.242+0000 7feaf7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:12.235 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.242+0000 7feaf7fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:28:12.235 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.242+0000 7feaf7fff640 1 -- 172.21.15.86:0/2190546550 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb100716e0 con 0x7feb100aa2a0 2024-03-29T11:28:12.235 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.242+0000 7feaf7fff640 10 monclient: _finish_auth 0 2024-03-29T11:28:12.235 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.242+0000 7feaf7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:28:12.235 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.242+0000 7feaf7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:12.235 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.242+0000 7feaf7fff640 10 client.?.objecter ms_handle_connect 0x7feb1013b3c0 2024-03-29T11:28:12.235 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.242+0000 7feaf7fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:12.235 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.242+0000 7feb1619d640 5 monclient: authenticate success, global_id 4459 2024-03-29T11:28:12.235 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.242+0000 7feb1619d640 10 monclient: _renew_subs 2024-03-29T11:28:12.235 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.242+0000 7feb1619d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:28:12.235 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.242+0000 7feb1619d640 1 -- 172.21.15.86:0/2190546550 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb1011f5d0 con 0x7feb100aa2a0 2024-03-29T11:28:12.236 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.243+0000 7feb1619d640 10 client.4459.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:12.236 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.243+0000 7feb1619d640 10 monclient: _renew_subs 2024-03-29T11:28:12.236 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.243+0000 7feaf7fff640 1 -- 172.21.15.86:0/2190546550 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7feb00001be0 con 0x7feb100aa2a0 2024-03-29T11:28:12.236 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.243+0000 7feb1619d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:28:12.236 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.243+0000 7feb1619d640 1 -- 172.21.15.86:0/2190546550 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7feb1006ec20 con 0x7feb100aa2a0 2024-03-29T11:28:12.236 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.243+0000 7feaf7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:12.236 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.243+0000 7feaf7fff640 1 -- 172.21.15.86:0/2190546550 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7feb00003790 con 0x7feb100aa2a0 2024-03-29T11:28:12.236 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.243+0000 7feaf7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:12.236 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.243+0000 7feaf7fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:28:12.236 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.243+0000 7feaf7fff640 10 monclient: dump: 2024-03-29T11:28:12.236 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:28:12.236 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:12.236 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:12.237 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:12.237 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:12.237 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:12.237 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:28:12.237 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:28:12.237 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:28:12.237 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:12.237 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.243+0000 7feb0d7fa640 4 set_mon_vals no callback set 2024-03-29T11:28:12.237 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.243+0000 7feb1619d640 1 librados: init done 2024-03-29T11:28:12.237 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.245+0000 7feb1619d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:12.237 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.245+0000 7feb1619d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:12.237 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.245+0000 7feb1619d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:28:12.238 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.245+0000 7feaf7fff640 1 -- 172.21.15.86:0/2190546550 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7feb00019760 con 0x7feb100aa2a0 2024-03-29T11:28:12.238 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.246+0000 7feaf7fff640 1 -- 172.21.15.86:0/2190546550 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7feb00083630 con 0x7feb100aa2a0 2024-03-29T11:28:12.238 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.246+0000 7feaf7fff640 10 client.4459.objecter ms_dispatch 0x7feb10000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:12.238 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.246+0000 7feaf7fff640 3 client.4459.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:12.238 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.246+0000 7feaf7fff640 3 client.4459.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:12.238 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.246+0000 7feaf7fff640 20 client.4459.objecter dump_active .. 0 homeless 2024-03-29T11:28:12.238 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.245+0000 7feb1619d640 1 -- 172.21.15.86:0/2190546550 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb10072400 con 0x7feb100aa2a0 2024-03-29T11:28:12.239 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.246+0000 7feb0effd640 10 monclient: get_auth_request con 0x7feafc06ddf0 auth_method 0 2024-03-29T11:28:12.239 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.248+0000 7feaf7fff640 10 client.4459.objecter ms_handle_connect 0x7feafc06ddf0 2024-03-29T11:28:12.243 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.254+0000 7feaf7fff640 1 -- 172.21.15.86:0/2190546550 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7feb0004ef10 con 0x7feb100aa2a0 2024-03-29T11:28:12.243 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.254+0000 7feaf7fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:12.243 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.254+0000 7feaf7fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:12.587 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.597+0000 7feb1619d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:12.587 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.597+0000 7feb1619d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:12.587 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.597+0000 7feb1619d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:28:12.587 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.597+0000 7feb1619d640 1 -- 172.21.15.86:0/2190546550 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feb10072090 con 0x7feb100aa2a0 2024-03-29T11:28:12.589 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.598+0000 7feaf7fff640 1 -- 172.21.15.86:0/2190546550 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7feb00050e40 con 0x7feb100aa2a0 2024-03-29T11:28:12.589 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.598+0000 7feaf7fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:12.589 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.598+0000 7feaf7fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:12.589 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:28:12.589 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:12.592 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.602+0000 7feb1619d640 10 librados: watch_flush enter 2024-03-29T11:28:12.592 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.602+0000 7feb1619d640 10 librados: watch_flush exit 2024-03-29T11:28:12.592 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.602+0000 7feb1619d640 20 client.4459.objecter shutdown clearing up homeless session... 2024-03-29T11:28:12.592 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.602+0000 7feb1619d640 10 client.4459.objecter successfully canceled tick 2024-03-29T11:28:12.592 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.602+0000 7feb1619d640 1 -- 172.21.15.86:0/2190546550 >> v1:172.21.15.196:6816/588110318 conn(0x7feafc06ddf0 legacy=0x7feafc0702b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:12.593 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.602+0000 7feb1619d640 10 monclient: shutdown 2024-03-29T11:28:12.593 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.602+0000 7feb1619d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:12.593 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.602+0000 7feb1619d640 1 -- 172.21.15.86:0/2190546550 >> v1:172.21.15.196:6789/0 conn(0x7feb100aa2a0 legacy=0x7feb10139ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:12.593 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.603+0000 7feb1619d640 1 -- 172.21.15.86:0/2190546550 shutdown_connections 2024-03-29T11:28:12.593 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.603+0000 7feb1619d640 1 -- 172.21.15.86:0/2190546550 >> 172.21.15.86:0/2190546550 conn(0x7feb100b35e0 msgr2=0x7feb100ac760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:12.593 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.603+0000 7feb1619d640 1 -- 172.21.15.86:0/2190546550 shutdown_connections 2024-03-29T11:28:12.593 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.603+0000 7feb1619d640 1 -- 172.21.15.86:0/2190546550 wait complete. 2024-03-29T11:28:12.593 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:12.604+0000 7feb1619d640 1 librados: shutdown 2024-03-29T11:28:12.609 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:19.416 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:19.562 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.573+0000 7efc5dad4640 10 monclient: get_monmap_and_config 2024-03-29T11:28:19.562 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.573+0000 7efc5dad4640 10 monclient: build_initial_monmap 2024-03-29T11:28:19.563 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.573+0000 7efc5dad4640 10 monclient: monmap: 2024-03-29T11:28:19.563 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:28:19.563 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:19.563 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:28:19.574093+0000 2024-03-29T11:28:19.563 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:28:19.574093+0000 2024-03-29T11:28:19.563 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:19.563 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:19.563 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:28:19.563 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:28:19.563 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:28:19.563 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:19.564 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.574+0000 7efc5dad4640 1 Processor -- start 2024-03-29T11:28:19.564 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.574+0000 7efc5dad4640 1 -- start start 2024-03-29T11:28:19.564 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.574+0000 7efc5dad4640 10 monclient: init 2024-03-29T11:28:19.565 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.575+0000 7efc5dad4640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:19.565 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.575+0000 7efc5dad4640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:28:19.565 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.575+0000 7efc5dad4640 10 monclient(hunting): picked mon.noname-a con 0x7efc581422b0 addr v1:172.21.15.86:6789/0 2024-03-29T11:28:19.565 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.575+0000 7efc577fe640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7efc581422b0 0x7efc5813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:43590/0 (socket says 172.21.15.86:43590) 2024-03-29T11:28:19.565 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.575+0000 7efc577fe640 1 -- 172.21.15.86:0/3952296386 learned_addr learned my addr 172.21.15.86:0/3952296386 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:28:19.567 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.575+0000 7efc5dad4640 10 monclient(hunting): picked mon.noname-b con 0x7efc58142690 addr v1:172.21.15.86:6790/0 2024-03-29T11:28:19.567 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.577+0000 7efc5dad4640 10 monclient(hunting): picked mon.noname-c con 0x7efc580b17d0 addr v1:172.21.15.196:6789/0 2024-03-29T11:28:19.567 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.577+0000 7efc5dad4640 1 -- 172.21.15.86:0/3952296386 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc58142c80 con 0x7efc581422b0 2024-03-29T11:28:19.567 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.577+0000 7efc5dad4640 1 -- 172.21.15.86:0/3952296386 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc580aae40 con 0x7efc580b17d0 2024-03-29T11:28:19.567 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.577+0000 7efc5dad4640 1 -- 172.21.15.86:0/3952296386 --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc5813fbd0 con 0x7efc58142690 2024-03-29T11:28:19.568 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.577+0000 7efc5dad4640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:19.568 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.577+0000 7efc5dad4640 10 monclient(hunting): authenticate will time out at 1323.895952s 2024-03-29T11:28:19.568 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.578+0000 7efc567fc640 1 -- 172.21.15.86:0/3952296386 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 21956800 0 0) 0x7efc58142c80 con 0x7efc581422b0 2024-03-29T11:28:19.568 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.578+0000 7efc567fc640 10 monclient(hunting): _negotiate 2024-03-29T11:28:19.569 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.578+0000 7efc567fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:19.569 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.578+0000 7efc567fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:19.569 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.578+0000 7efc567fc640 10 monclient(hunting): my global_id is 4482 2024-03-29T11:28:19.569 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.578+0000 7efc567fc640 1 -- 172.21.15.86:0/3952296386 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc40003520 con 0x7efc581422b0 2024-03-29T11:28:19.571 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.579+0000 7efc567fc640 1 -- 172.21.15.86:0/3952296386 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1002100732 0 0) 0x7efc40003520 con 0x7efc581422b0 2024-03-29T11:28:19.571 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.579+0000 7efc567fc640 1 -- 172.21.15.86:0/3952296386 >> v1:172.21.15.86:6790/0 conn(0x7efc58142690 legacy=0x7efc580b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:19.571 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.579+0000 7efc567fc640 1 -- 172.21.15.86:0/3952296386 >> v1:172.21.15.196:6789/0 conn(0x7efc580b17d0 legacy=0x7efc580b2ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:19.575 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.581+0000 7efc567fc640 10 monclient: _finish_hunting 0 2024-03-29T11:28:19.575 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.581+0000 7efc567fc640 1 monclient: found mon.noname-a 2024-03-29T11:28:19.575 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.581+0000 7efc567fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:19.575 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.581+0000 7efc567fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.86:6789/0 2024-03-29T11:28:19.575 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.581+0000 7efc567fc640 1 -- 172.21.15.86:0/3952296386 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc58150930 con 0x7efc581422b0 2024-03-29T11:28:19.575 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.581+0000 7efc567fc640 10 monclient: _finish_auth 0 2024-03-29T11:28:19.575 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.581+0000 7efc567fc640 10 monclient: _check_auth_tickets 2024-03-29T11:28:19.575 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.581+0000 7efc567fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:19.575 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.581+0000 7efc567fc640 1 -- 172.21.15.86:0/3952296386 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7efc48001690 con 0x7efc581422b0 2024-03-29T11:28:19.575 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.581+0000 7efc567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:19.575 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.581+0000 7efc567fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:19.575 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.581+0000 7efc567fc640 10 monclient: dump: 2024-03-29T11:28:19.576 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:28:19.576 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:19.576 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:19.576 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:19.576 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:19.576 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:19.576 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:28:19.576 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:28:19.576 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:28:19.576 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:19.576 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.582+0000 7efc5dad4640 5 monclient: authenticate success, global_id 4482 2024-03-29T11:28:19.577 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.582+0000 7efc5dad4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:19.577 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.582+0000 7efc567fc640 1 -- 172.21.15.86:0/3952296386 <== mon.0 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc480017f0 con 0x7efc581422b0 2024-03-29T11:28:19.577 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.582+0000 7efc567fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:19.577 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.582+0000 7efc567fc640 1 -- 172.21.15.86:0/3952296386 <== mon.0 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7efc48001c40 con 0x7efc581422b0 2024-03-29T11:28:19.577 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.582+0000 7efc567fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:19.577 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.582+0000 7efc567fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:19.578 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.582+0000 7efc567fc640 10 monclient: dump: 2024-03-29T11:28:19.578 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:28:19.578 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:19.578 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:19.578 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:19.578 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:19.578 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:19.578 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:28:19.578 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:28:19.578 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:28:19.578 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:19.578 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.582+0000 7efc5dad4640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:19.578 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.582+0000 7efc5dad4640 4 set_mon_vals no callback set 2024-03-29T11:28:19.579 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.583+0000 7efc5dad4640 10 monclient: shutdown 2024-03-29T11:28:19.579 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.583+0000 7efc5dad4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:19.579 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.583+0000 7efc5dad4640 1 -- 172.21.15.86:0/3952296386 >> v1:172.21.15.86:6789/0 conn(0x7efc581422b0 legacy=0x7efc5813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:19.579 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.583+0000 7efc5dad4640 1 -- 172.21.15.86:0/3952296386 shutdown_connections 2024-03-29T11:28:19.579 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.583+0000 7efc5dad4640 1 -- 172.21.15.86:0/3952296386 >> 172.21.15.86:0/3952296386 conn(0x7efc58066a00 msgr2=0x7efc58066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:19.579 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.584+0000 7efc5dad4640 1 -- 172.21.15.86:0/3952296386 shutdown_connections 2024-03-29T11:28:19.579 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.584+0000 7efc5dad4640 1 -- 172.21.15.86:0/3952296386 wait complete. 2024-03-29T11:28:19.579 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.585+0000 7efc5dad4640 10 monclient: build_initial_monmap 2024-03-29T11:28:19.579 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.585+0000 7efc5dad4640 10 monclient: monmap: 2024-03-29T11:28:19.579 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:28:19.579 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:19.579 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:28:19.579 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:28:19.580 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:19.580 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:19.580 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:28:19.580 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:28:19.580 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:28:19.580 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:19.580 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.585+0000 7efc5dad4640 1 librados: starting msgr at 2024-03-29T11:28:19.580 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.585+0000 7efc5dad4640 1 librados: starting objecter 2024-03-29T11:28:19.587 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.585+0000 7efc5dad4640 1 Processor -- start 2024-03-29T11:28:19.587 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.586+0000 7efc5dad4640 1 -- start start 2024-03-29T11:28:19.587 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.586+0000 7efc5dad4640 1 librados: setting wanted keys 2024-03-29T11:28:19.588 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.586+0000 7efc5dad4640 1 librados: calling monclient init 2024-03-29T11:28:19.588 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.586+0000 7efc5dad4640 10 monclient: init 2024-03-29T11:28:19.588 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.587+0000 7efc5dad4640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:19.588 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.587+0000 7efc5dad4640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:19.588 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.587+0000 7efc5dad4640 10 monclient(hunting): picked mon.noname-c con 0x7efc580b17d0 addr v1:172.21.15.86:6790/0 2024-03-29T11:28:19.588 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.590+0000 7efc5dad4640 10 monclient(hunting): picked mon.noname-a con 0x7efc581422b0 addr v1:172.21.15.86:6789/0 2024-03-29T11:28:19.588 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.590+0000 7efc577fe640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7efc580b17d0 0x7efc580ac2f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:53139/0 (socket says 172.21.15.86:53139) 2024-03-29T11:28:19.588 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.591+0000 7efc5dad4640 10 monclient(hunting): picked mon.noname-b con 0x7efc58142690 addr v1:172.21.15.196:6789/0 2024-03-29T11:28:19.588 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.591+0000 7efc5dad4640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc580acdc0 con 0x7efc581422b0 2024-03-29T11:28:19.588 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.591+0000 7efc5dad4640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc580adc90 con 0x7efc58142690 2024-03-29T11:28:19.588 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.591+0000 7efc5dad4640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efc580adf70 con 0x7efc580b17d0 2024-03-29T11:28:19.588 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.591+0000 7efc5dad4640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:19.588 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.591+0000 7efc5dad4640 10 monclient(hunting): authenticate will time out at 1323.909862s 2024-03-29T11:28:19.588 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.591+0000 7efc56ffd640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7efc581422b0 0x7efc580b5640 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:45032/0 (socket says 172.21.15.86:45032) 2024-03-29T11:28:19.589 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.592+0000 7efc56ffd640 1 -- 172.21.15.86:0/1379940280 learned_addr learned my addr 172.21.15.86:0/1379940280 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:28:19.589 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.592+0000 7efc3ffff640 10 client.?.objecter ms_handle_connect 0x7efc581422b0 2024-03-29T11:28:19.589 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.592+0000 7efc3ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:19.589 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.592+0000 7efc3ffff640 10 client.?.objecter ms_handle_connect 0x7efc580b17d0 2024-03-29T11:28:19.589 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.592+0000 7efc3ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:19.589 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.592+0000 7efc3ffff640 1 -- 172.21.15.86:0/1379940280 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 453237968 0 0) 0x7efc580adf70 con 0x7efc580b17d0 2024-03-29T11:28:19.589 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.592+0000 7efc3ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:19.589 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.592+0000 7efc3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:19.589 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.592+0000 7efc3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:19.589 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.592+0000 7efc3ffff640 10 monclient(hunting): my global_id is 4553 2024-03-29T11:28:19.589 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.592+0000 7efc3ffff640 1 -- 172.21.15.86:0/1379940280 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efc44004eb0 con 0x7efc580b17d0 2024-03-29T11:28:19.589 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.593+0000 7efc3ffff640 1 -- 172.21.15.86:0/1379940280 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4280017619 0 0) 0x7efc44004eb0 con 0x7efc580b17d0 2024-03-29T11:28:19.589 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.593+0000 7efc3ffff640 1 -- 172.21.15.86:0/1379940280 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7efc580adf70 con 0x7efc580b17d0 2024-03-29T11:28:19.589 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.593+0000 7efc3ffff640 1 -- 172.21.15.86:0/1379940280 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7efc48001b00 con 0x7efc580b17d0 2024-03-29T11:28:19.590 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.593+0000 7efc3ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:19.590 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.593+0000 7efc3ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:19.590 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.593+0000 7efc3ffff640 10 monclient(hunting): dump: 2024-03-29T11:28:19.590 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:28:19.590 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:19.590 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:19.590 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:19.590 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:19.590 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:19.590 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:28:19.590 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:28:19.590 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:28:19.590 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:19.590 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.593+0000 7efc3ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:19.591 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.593+0000 7efc3ffff640 1 -- 172.21.15.86:0/1379940280 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2196763872 0 0) 0x7efc580adf70 con 0x7efc580b17d0 2024-03-29T11:28:19.591 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.593+0000 7efc3ffff640 1 -- 172.21.15.86:0/1379940280 >> v1:172.21.15.196:6789/0 conn(0x7efc58142690 legacy=0x7efc580b5d50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:19.591 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.594+0000 7efc3ffff640 1 -- 172.21.15.86:0/1379940280 >> v1:172.21.15.86:6789/0 conn(0x7efc581422b0 legacy=0x7efc580b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:19.591 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.594+0000 7efc3ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:28:19.591 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.594+0000 7efc3ffff640 1 monclient: found mon.c 2024-03-29T11:28:19.591 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.594+0000 7efc3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:19.591 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.594+0000 7efc3ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:28:19.591 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.594+0000 7efc3ffff640 1 -- 172.21.15.86:0/1379940280 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc580abdd0 con 0x7efc580b17d0 2024-03-29T11:28:19.591 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.594+0000 7efc3ffff640 10 monclient: _finish_auth 0 2024-03-29T11:28:19.591 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.594+0000 7efc3ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:28:19.591 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.594+0000 7efc3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:19.591 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.594+0000 7efc5dad4640 5 monclient: authenticate success, global_id 4553 2024-03-29T11:28:19.591 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.594+0000 7efc5dad4640 10 monclient: _renew_subs 2024-03-29T11:28:19.591 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.594+0000 7efc5dad4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:28:19.592 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.594+0000 7efc5dad4640 1 -- 172.21.15.86:0/1379940280 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc580aca00 con 0x7efc580b17d0 2024-03-29T11:28:19.592 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.594+0000 7efc5dad4640 10 client.4553.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:19.592 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.594+0000 7efc5dad4640 10 monclient: _renew_subs 2024-03-29T11:28:19.592 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.594+0000 7efc5dad4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:28:19.592 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.594+0000 7efc5dad4640 1 -- 172.21.15.86:0/1379940280 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7efc581e2fc0 con 0x7efc580b17d0 2024-03-29T11:28:19.592 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.594+0000 7efc5dad4640 1 librados: init done 2024-03-29T11:28:19.592 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.595+0000 7efc3ffff640 1 -- 172.21.15.86:0/1379940280 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7efc48001d20 con 0x7efc580b17d0 2024-03-29T11:28:19.592 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.595+0000 7efc3ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:19.592 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.595+0000 7efc3ffff640 1 -- 172.21.15.86:0/1379940280 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7efc48004d90 con 0x7efc580b17d0 2024-03-29T11:28:19.592 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.595+0000 7efc3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:19.592 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.595+0000 7efc3ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:19.592 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.595+0000 7efc3ffff640 10 monclient: dump: 2024-03-29T11:28:19.592 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:28:19.592 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:19.592 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:19.593 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:19.593 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:19.593 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:19.593 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:28:19.593 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:28:19.593 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:28:19.593 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:19.593 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.595+0000 7efc557fa640 4 set_mon_vals no callback set 2024-03-29T11:28:19.593 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.596+0000 7efc3ffff640 1 -- 172.21.15.86:0/1379940280 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7efc4801ad60 con 0x7efc580b17d0 2024-03-29T11:28:19.593 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.596+0000 7efc5dad4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:19.593 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.597+0000 7efc3ffff640 1 -- 172.21.15.86:0/1379940280 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7efc48084bb0 con 0x7efc580b17d0 2024-03-29T11:28:19.593 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.597+0000 7efc3ffff640 10 client.4553.objecter ms_dispatch 0x7efc58000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:19.593 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.597+0000 7efc3ffff640 3 client.4553.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:19.593 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.597+0000 7efc3ffff640 3 client.4553.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:19.593 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.597+0000 7efc3ffff640 20 client.4553.objecter dump_active .. 0 homeless 2024-03-29T11:28:19.594 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.597+0000 7efc5dad4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:19.594 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.597+0000 7efc5dad4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:28:19.594 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.597+0000 7efc5dad4640 1 -- 172.21.15.86:0/1379940280 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc38005080 con 0x7efc580b17d0 2024-03-29T11:28:19.594 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.597+0000 7efc56ffd640 10 monclient: get_auth_request con 0x7efc4406db40 auth_method 0 2024-03-29T11:28:19.594 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.599+0000 7efc3ffff640 10 client.4553.objecter ms_handle_connect 0x7efc4406db40 2024-03-29T11:28:19.596 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.607+0000 7efc3ffff640 1 -- 172.21.15.86:0/1379940280 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7efc480504f0 con 0x7efc580b17d0 2024-03-29T11:28:19.596 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.607+0000 7efc3ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:19.597 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.607+0000 7efc3ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:19.930 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:28:19.931 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:19.931 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.938+0000 7efc5dad4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:19.931 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.938+0000 7efc5dad4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:19.931 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.938+0000 7efc5dad4640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:28:19.931 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.938+0000 7efc5dad4640 1 -- 172.21.15.86:0/1379940280 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7efc38005330 con 0x7efc580b17d0 2024-03-29T11:28:19.931 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.938+0000 7efc3ffff640 1 -- 172.21.15.86:0/1379940280 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7efc48084260 con 0x7efc580b17d0 2024-03-29T11:28:19.931 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.938+0000 7efc3ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:19.931 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.938+0000 7efc3ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:19.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.942+0000 7efc5dad4640 10 librados: watch_flush enter 2024-03-29T11:28:19.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.942+0000 7efc5dad4640 10 librados: watch_flush exit 2024-03-29T11:28:19.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.942+0000 7efc5dad4640 20 client.4553.objecter shutdown clearing up homeless session... 2024-03-29T11:28:19.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.942+0000 7efc5dad4640 10 client.4553.objecter successfully canceled tick 2024-03-29T11:28:19.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.944+0000 7efc5dad4640 1 -- 172.21.15.86:0/1379940280 >> v1:172.21.15.196:6816/588110318 conn(0x7efc4406db40 legacy=0x7efc44070000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:19.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.944+0000 7efc5dad4640 10 monclient: shutdown 2024-03-29T11:28:19.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.944+0000 7efc5dad4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:19.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.944+0000 7efc5dad4640 1 -- 172.21.15.86:0/1379940280 >> v1:172.21.15.86:6790/0 conn(0x7efc580b17d0 legacy=0x7efc580ac2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:19.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.944+0000 7efc5dad4640 1 -- 172.21.15.86:0/1379940280 shutdown_connections 2024-03-29T11:28:19.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.944+0000 7efc5dad4640 1 -- 172.21.15.86:0/1379940280 >> 172.21.15.86:0/1379940280 conn(0x7efc58066a00 msgr2=0x7efc5813c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:19.936 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.946+0000 7efc5dad4640 1 -- 172.21.15.86:0/1379940280 shutdown_connections 2024-03-29T11:28:19.936 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.946+0000 7efc5dad4640 1 -- 172.21.15.86:0/1379940280 wait complete. 2024-03-29T11:28:19.936 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:19.947+0000 7efc5dad4640 1 librados: shutdown 2024-03-29T11:28:19.955 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:26.763 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:26.904 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.914+0000 7f60fb412640 10 monclient: get_monmap_and_config 2024-03-29T11:28:26.904 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.914+0000 7f60fb412640 10 monclient: build_initial_monmap 2024-03-29T11:28:26.904 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.914+0000 7f60fb412640 10 monclient: monmap: 2024-03-29T11:28:26.904 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:28:26.904 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:26.905 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:28:26.915454+0000 2024-03-29T11:28:26.905 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:28:26.915454+0000 2024-03-29T11:28:26.905 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:26.905 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:26.905 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:28:26.905 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:28:26.905 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:28:26.905 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:26.906 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.915+0000 7f60fb412640 1 Processor -- start 2024-03-29T11:28:26.906 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.915+0000 7f60fb412640 1 -- start start 2024-03-29T11:28:26.906 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.915+0000 7f60fb412640 10 monclient: init 2024-03-29T11:28:26.906 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.916+0000 7f60fb412640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:26.906 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.916+0000 7f60fb412640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:26.906 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.916+0000 7f60fb412640 10 monclient(hunting): picked mon.noname-b con 0x7f60f40b2800 addr v1:172.21.15.86:6790/0 2024-03-29T11:28:26.908 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.917+0000 7f60fa410640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f60f40b2800 0x7f60f4142620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:38034/0 (socket says 172.21.15.86:38034) 2024-03-29T11:28:26.908 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.917+0000 7f60fa410640 1 -- 172.21.15.86:0/565837566 learned_addr learned my addr 172.21.15.86:0/565837566 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:28:26.908 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.917+0000 7f60fb412640 10 monclient(hunting): picked mon.noname-a con 0x7f60f40a9c00 addr v1:172.21.15.86:6789/0 2024-03-29T11:28:26.908 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.917+0000 7f60fb412640 10 monclient(hunting): picked mon.noname-c con 0x7f60f40b38c0 addr v1:172.21.15.196:6789/0 2024-03-29T11:28:26.908 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.917+0000 7f60fb412640 1 -- 172.21.15.86:0/565837566 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60f4141eb0 con 0x7f60f40a9c00 2024-03-29T11:28:26.908 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.917+0000 7f60fb412640 1 -- 172.21.15.86:0/565837566 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60f40b22d0 con 0x7f60f40b38c0 2024-03-29T11:28:26.908 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.917+0000 7f60fb412640 1 -- 172.21.15.86:0/565837566 --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60f4142d30 con 0x7f60f40b2800 2024-03-29T11:28:26.908 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.917+0000 7f60fb412640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:26.909 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.917+0000 7f60fb412640 10 monclient(hunting): authenticate will time out at 1331.235854s 2024-03-29T11:28:26.909 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.918+0000 7f60f940e640 1 -- 172.21.15.86:0/565837566 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2591249100 0 0) 0x7f60f4142d30 con 0x7f60f40b2800 2024-03-29T11:28:26.909 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.918+0000 7f60f940e640 10 monclient(hunting): _negotiate 2024-03-29T11:28:26.909 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.918+0000 7f60f940e640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:26.909 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.918+0000 7f60f940e640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:26.909 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.918+0000 7f60f940e640 10 monclient(hunting): my global_id is 4556 2024-03-29T11:28:26.909 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.918+0000 7f60f940e640 1 -- 172.21.15.86:0/565837566 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60e0003520 con 0x7f60f40b2800 2024-03-29T11:28:26.909 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.918+0000 7f60f940e640 1 -- 172.21.15.86:0/565837566 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4117711553 0 0) 0x7f60e0003520 con 0x7f60f40b2800 2024-03-29T11:28:26.909 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.918+0000 7f60f940e640 1 -- 172.21.15.86:0/565837566 >> v1:172.21.15.196:6789/0 conn(0x7f60f40b38c0 legacy=0x7f60f414d7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:26.913 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.921+0000 7f60f940e640 1 -- 172.21.15.86:0/565837566 >> v1:172.21.15.86:6789/0 conn(0x7f60f40a9c00 legacy=0x7f60f413bb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:26.914 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.921+0000 7f60f940e640 10 monclient: _finish_hunting 0 2024-03-29T11:28:26.914 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.921+0000 7f60f940e640 1 monclient: found mon.noname-b 2024-03-29T11:28:26.914 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.921+0000 7f60f940e640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:26.914 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.921+0000 7f60f940e640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:28:26.914 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.921+0000 7f60f940e640 1 -- 172.21.15.86:0/565837566 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60f40aaf90 con 0x7f60f40b2800 2024-03-29T11:28:26.914 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.921+0000 7f60f940e640 10 monclient: _finish_auth 0 2024-03-29T11:28:26.914 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.921+0000 7f60f940e640 10 monclient: _check_auth_tickets 2024-03-29T11:28:26.914 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.921+0000 7f60f940e640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:26.914 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.921+0000 7f60f940e640 1 -- 172.21.15.86:0/565837566 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f60f0001690 con 0x7f60f40b2800 2024-03-29T11:28:26.914 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.921+0000 7f60f940e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:26.914 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.921+0000 7f60f940e640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:26.914 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.921+0000 7f60f940e640 10 monclient: dump: 2024-03-29T11:28:26.914 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:28:26.914 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:26.915 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:26.915 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:26.915 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:26.915 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:26.915 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:28:26.915 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:28:26.915 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:28:26.915 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:26.915 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.921+0000 7f60fb412640 5 monclient: authenticate success, global_id 4556 2024-03-29T11:28:26.915 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.921+0000 7f60fb412640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:26.915 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.922+0000 7f60f940e640 1 -- 172.21.15.86:0/565837566 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60f00017f0 con 0x7f60f40b2800 2024-03-29T11:28:26.915 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.922+0000 7f60f940e640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:26.915 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.922+0000 7f60f940e640 1 -- 172.21.15.86:0/565837566 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f60f0001c00 con 0x7f60f40b2800 2024-03-29T11:28:26.915 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.922+0000 7f60f940e640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:26.916 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.922+0000 7f60f940e640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:26.916 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.922+0000 7f60f940e640 10 monclient: dump: 2024-03-29T11:28:26.916 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:28:26.916 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:26.916 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:26.916 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:26.916 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:26.916 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:26.916 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:28:26.916 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:28:26.916 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:28:26.916 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:26.916 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.922+0000 7f60fb412640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:26.916 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.922+0000 7f60fb412640 4 set_mon_vals no callback set 2024-03-29T11:28:26.916 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.923+0000 7f60fb412640 10 monclient: shutdown 2024-03-29T11:28:26.917 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.923+0000 7f60fb412640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:26.917 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.923+0000 7f60fb412640 1 -- 172.21.15.86:0/565837566 >> v1:172.21.15.86:6790/0 conn(0x7f60f40b2800 legacy=0x7f60f4142620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:26.917 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.923+0000 7f60fb412640 1 -- 172.21.15.86:0/565837566 shutdown_connections 2024-03-29T11:28:26.917 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.923+0000 7f60fb412640 1 -- 172.21.15.86:0/565837566 >> 172.21.15.86:0/565837566 conn(0x7f60f4066a00 msgr2=0x7f60f4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:26.917 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.923+0000 7f60fb412640 1 -- 172.21.15.86:0/565837566 shutdown_connections 2024-03-29T11:28:26.917 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.923+0000 7f60fb412640 1 -- 172.21.15.86:0/565837566 wait complete. 2024-03-29T11:28:26.917 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.924+0000 7f60fb412640 10 monclient: build_initial_monmap 2024-03-29T11:28:26.918 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.924+0000 7f60fb412640 10 monclient: monmap: 2024-03-29T11:28:26.918 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:28:26.918 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:26.918 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:28:26.918 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:28:26.918 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:26.918 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:26.918 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:28:26.918 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:28:26.918 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:28:26.918 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:26.920 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.924+0000 7f60fb412640 1 librados: starting msgr at 2024-03-29T11:28:26.920 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.924+0000 7f60fb412640 1 librados: starting objecter 2024-03-29T11:28:26.920 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.924+0000 7f60fb412640 1 Processor -- start 2024-03-29T11:28:26.920 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.924+0000 7f60fb412640 1 -- start start 2024-03-29T11:28:26.920 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.924+0000 7f60fb412640 1 librados: setting wanted keys 2024-03-29T11:28:26.920 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.924+0000 7f60fb412640 1 librados: calling monclient init 2024-03-29T11:28:26.920 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.924+0000 7f60fb412640 10 monclient: init 2024-03-29T11:28:26.920 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.925+0000 7f60fb412640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:26.920 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.925+0000 7f60fb412640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:28:26.920 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.925+0000 7f60fb412640 10 monclient(hunting): picked mon.noname-c con 0x7f60f40a9c00 addr v1:172.21.15.86:6790/0 2024-03-29T11:28:26.920 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.927+0000 7f60fb412640 10 monclient(hunting): picked mon.noname-a con 0x7f60f40b2800 addr v1:172.21.15.86:6789/0 2024-03-29T11:28:26.920 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.928+0000 7f60fa410640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f60f40a9c00 0x7f60f4145660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:38634/0 (socket says 172.21.15.86:38634) 2024-03-29T11:28:26.920 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.928+0000 7f60fb412640 10 monclient(hunting): picked mon.noname-b con 0x7f60f40b38c0 addr v1:172.21.15.196:6789/0 2024-03-29T11:28:26.921 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.928+0000 7f60fb412640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60f4145ee0 con 0x7f60f40b2800 2024-03-29T11:28:26.921 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.928+0000 7f60fa410640 1 -- 172.21.15.86:0/1313998131 learned_addr learned my addr 172.21.15.86:0/1313998131 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:28:26.921 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.928+0000 7f60fb412640 1 -- 172.21.15.86:0/1313998131 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60f4136780 con 0x7f60f40b38c0 2024-03-29T11:28:26.921 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.928+0000 7f60fb412640 1 -- 172.21.15.86:0/1313998131 --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60f4136910 con 0x7f60f40a9c00 2024-03-29T11:28:26.921 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.928+0000 7f60fb412640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:26.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.928+0000 7f60eaffd640 10 client.?.objecter ms_handle_connect 0x7f60f40a9c00 2024-03-29T11:28:26.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.928+0000 7f60eaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:26.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.930+0000 7f60eaffd640 1 -- 172.21.15.86:0/1313998131 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1829097728 0 0) 0x7f60f4136910 con 0x7f60f40a9c00 2024-03-29T11:28:26.928 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.930+0000 7f60fb412640 10 monclient(hunting): authenticate will time out at 1331.248951s 2024-03-29T11:28:26.929 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.931+0000 7f60eaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:26.929 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.931+0000 7f60eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:26.929 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.931+0000 7f60eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:26.929 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.931+0000 7f60eaffd640 10 monclient(hunting): my global_id is 4559 2024-03-29T11:28:26.929 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.931+0000 7f60eaffd640 1 -- 172.21.15.86:0/1313998131 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60d8004eb0 con 0x7f60f40a9c00 2024-03-29T11:28:26.929 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.931+0000 7f60eaffd640 10 client.?.objecter ms_handle_connect 0x7f60f40b2800 2024-03-29T11:28:26.929 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.931+0000 7f60eaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:26.929 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.931+0000 7f60eaffd640 1 -- 172.21.15.86:0/1313998131 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3943532711 0 0) 0x7f60f4145ee0 con 0x7f60f40b2800 2024-03-29T11:28:26.929 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.931+0000 7f60eaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:26.929 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.931+0000 7f60eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:26.929 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.931+0000 7f60eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:26.929 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.931+0000 7f60eaffd640 10 monclient(hunting): my global_id is 4491 2024-03-29T11:28:26.929 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.931+0000 7f60eaffd640 1 -- 172.21.15.86:0/1313998131 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60f4136910 con 0x7f60f40b2800 2024-03-29T11:28:26.929 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.932+0000 7f60eaffd640 1 -- 172.21.15.86:0/1313998131 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2864094867 0 0) 0x7f60d8004eb0 con 0x7f60f40a9c00 2024-03-29T11:28:26.930 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.932+0000 7f60eaffd640 1 -- 172.21.15.86:0/1313998131 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f60f4145ee0 con 0x7f60f40a9c00 2024-03-29T11:28:26.930 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.932+0000 7f60eaffd640 1 -- 172.21.15.86:0/1313998131 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f60f0001c00 con 0x7f60f40a9c00 2024-03-29T11:28:26.930 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.932+0000 7f60eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:26.930 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.932+0000 7f60eaffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:26.930 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.932+0000 7f60eaffd640 10 monclient(hunting): dump: 2024-03-29T11:28:26.930 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:28:26.930 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:26.930 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:26.930 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:26.930 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:26.930 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:26.930 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:28:26.930 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:28:26.930 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:28:26.931 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:26.931 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.932+0000 7f60eaffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:26.931 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.932+0000 7f60eaffd640 1 -- 172.21.15.86:0/1313998131 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 843169810 0 0) 0x7f60f4136910 con 0x7f60f40b2800 2024-03-29T11:28:26.931 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.932+0000 7f60eaffd640 1 -- 172.21.15.86:0/1313998131 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f60d8004eb0 con 0x7f60f40b2800 2024-03-29T11:28:26.931 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.932+0000 7f60eaffd640 10 client.?.objecter ms_handle_connect 0x7f60f40b38c0 2024-03-29T11:28:26.931 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.932+0000 7f60eaffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:26.931 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.932+0000 7f60eaffd640 1 -- 172.21.15.86:0/1313998131 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f60e40018b0 con 0x7f60f40b2800 2024-03-29T11:28:26.931 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.932+0000 7f60eaffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:26.931 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.932+0000 7f60eaffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:26.932 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.932+0000 7f60eaffd640 10 monclient(hunting): dump: 2024-03-29T11:28:26.932 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:28:26.932 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:26.932 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:26.932 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:26.932 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:26.932 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:26.932 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:28:26.932 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:28:26.932 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:28:26.932 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:26.932 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.932+0000 7f60eaffd640 1 -- 172.21.15.86:0/1313998131 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2671271790 0 0) 0x7f60f4136780 con 0x7f60f40b38c0 2024-03-29T11:28:26.932 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.932+0000 7f60eaffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:26.932 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.932+0000 7f60eaffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:26.932 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.933+0000 7f60eaffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:26.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.933+0000 7f60eaffd640 10 monclient(hunting): my global_id is 4462 2024-03-29T11:28:26.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.933+0000 7f60eaffd640 1 -- 172.21.15.86:0/1313998131 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f60f4136910 con 0x7f60f40b38c0 2024-03-29T11:28:26.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.933+0000 7f60eaffd640 1 -- 172.21.15.86:0/1313998131 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4096653885 0 0) 0x7f60f4145ee0 con 0x7f60f40a9c00 2024-03-29T11:28:26.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.933+0000 7f60eaffd640 1 -- 172.21.15.86:0/1313998131 >> v1:172.21.15.196:6789/0 conn(0x7f60f40b38c0 legacy=0x7f60f41d5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:26.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.933+0000 7f60eaffd640 1 -- 172.21.15.86:0/1313998131 >> v1:172.21.15.86:6789/0 conn(0x7f60f40b2800 legacy=0x7f60f413ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:26.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.933+0000 7f60eaffd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:26.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.933+0000 7f60eaffd640 1 monclient: found mon.c 2024-03-29T11:28:26.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.933+0000 7f60eaffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:26.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.933+0000 7f60eaffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:28:26.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.933+0000 7f60eaffd640 1 -- 172.21.15.86:0/1313998131 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60f413f380 con 0x7f60f40a9c00 2024-03-29T11:28:26.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.933+0000 7f60eaffd640 10 monclient: _finish_auth 0 2024-03-29T11:28:26.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.933+0000 7f60eaffd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:26.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.933+0000 7f60eaffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:26.933 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.933+0000 7f60fb412640 5 monclient: authenticate success, global_id 4559 2024-03-29T11:28:26.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.934+0000 7f60fb412640 10 monclient: _renew_subs 2024-03-29T11:28:26.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.934+0000 7f60fb412640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:28:26.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.934+0000 7f60fb412640 1 -- 172.21.15.86:0/1313998131 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60f413f630 con 0x7f60f40a9c00 2024-03-29T11:28:26.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.934+0000 7f60fb412640 10 client.4559.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:26.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.934+0000 7f60fb412640 10 monclient: _renew_subs 2024-03-29T11:28:26.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.934+0000 7f60fb412640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:28:26.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.934+0000 7f60fb412640 1 -- 172.21.15.86:0/1313998131 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f60f4145240 con 0x7f60f40a9c00 2024-03-29T11:28:26.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.934+0000 7f60fb412640 1 librados: init done 2024-03-29T11:28:26.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.935+0000 7f60eaffd640 1 -- 172.21.15.86:0/1313998131 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f60f0004b00 con 0x7f60f40a9c00 2024-03-29T11:28:26.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.935+0000 7f60eaffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:26.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.935+0000 7f60eaffd640 1 -- 172.21.15.86:0/1313998131 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f60f0004e10 con 0x7f60f40a9c00 2024-03-29T11:28:26.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.935+0000 7f60f8c0d640 4 set_mon_vals no callback set 2024-03-29T11:28:26.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.935+0000 7f60eaffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:26.934 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.935+0000 7f60eaffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:26.935 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.936+0000 7f60eaffd640 10 monclient: dump: 2024-03-29T11:28:26.935 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:28:26.935 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:26.935 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:26.935 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:26.935 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:26.935 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:26.935 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:28:26.935 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:28:26.935 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:28:26.935 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:26.935 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.936+0000 7f60fb412640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:26.935 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.936+0000 7f60fb412640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:26.935 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.936+0000 7f60fb412640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:28:26.936 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.936+0000 7f60fb412640 1 -- 172.21.15.86:0/1313998131 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60dc005080 con 0x7f60f40a9c00 2024-03-29T11:28:26.936 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.937+0000 7f60eaffd640 1 -- 172.21.15.86:0/1313998131 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f60f0004f70 con 0x7f60f40a9c00 2024-03-29T11:28:26.936 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.938+0000 7f60eaffd640 1 -- 172.21.15.86:0/1313998131 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f60f0084c20 con 0x7f60f40a9c00 2024-03-29T11:28:26.936 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.938+0000 7f60eaffd640 10 client.4559.objecter ms_dispatch 0x7f60f4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:26.936 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.938+0000 7f60eaffd640 3 client.4559.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:26.936 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.938+0000 7f60eaffd640 3 client.4559.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:26.936 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.938+0000 7f60eaffd640 20 client.4559.objecter dump_active .. 0 homeless 2024-03-29T11:28:26.936 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.938+0000 7f60f9c0f640 10 monclient: get_auth_request con 0x7f60d806dfc0 auth_method 0 2024-03-29T11:28:26.936 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.939+0000 7f60eaffd640 10 client.4559.objecter ms_handle_connect 0x7f60d806dfc0 2024-03-29T11:28:26.937 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.946+0000 7f60eaffd640 1 -- 172.21.15.86:0/1313998131 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f60f0050500 con 0x7f60f40a9c00 2024-03-29T11:28:26.937 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.946+0000 7f60eaffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:26.937 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:26.946+0000 7f60eaffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:27.245 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:27.254+0000 7f60fb412640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:27.245 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:27.254+0000 7f60fb412640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:27.245 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:27.254+0000 7f60fb412640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:28:27.246 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:27.254+0000 7f60fb412640 1 -- 172.21.15.86:0/1313998131 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60dc005330 con 0x7f60f40a9c00 2024-03-29T11:28:27.246 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:27.256+0000 7f60eaffd640 1 -- 172.21.15.86:0/1313998131 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f60f0052430 con 0x7f60f40a9c00 2024-03-29T11:28:27.246 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:27.256+0000 7f60eaffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:27.246 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:27.256+0000 7f60eaffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:27.246 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:28:27.247 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:27.249 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:27.259+0000 7f60fb412640 10 librados: watch_flush enter 2024-03-29T11:28:27.249 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:27.259+0000 7f60fb412640 10 librados: watch_flush exit 2024-03-29T11:28:27.249 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:27.259+0000 7f60fb412640 20 client.4559.objecter shutdown clearing up homeless session... 2024-03-29T11:28:27.249 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:27.259+0000 7f60fb412640 10 client.4559.objecter successfully canceled tick 2024-03-29T11:28:27.250 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:27.260+0000 7f60fb412640 1 -- 172.21.15.86:0/1313998131 >> v1:172.21.15.196:6816/588110318 conn(0x7f60d806dfc0 legacy=0x7f60d8070480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:27.250 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:27.260+0000 7f60fb412640 10 monclient: shutdown 2024-03-29T11:28:27.250 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:27.260+0000 7f60fb412640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:27.251 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:27.260+0000 7f60fb412640 1 -- 172.21.15.86:0/1313998131 >> v1:172.21.15.86:6790/0 conn(0x7f60f40a9c00 legacy=0x7f60f4145660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:27.251 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:27.260+0000 7f60fb412640 1 -- 172.21.15.86:0/1313998131 shutdown_connections 2024-03-29T11:28:27.251 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:27.260+0000 7f60fb412640 1 -- 172.21.15.86:0/1313998131 >> 172.21.15.86:0/1313998131 conn(0x7f60f4066a00 msgr2=0x7f60f413b9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:27.251 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:27.261+0000 7f60fb412640 1 -- 172.21.15.86:0/1313998131 shutdown_connections 2024-03-29T11:28:27.251 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:27.261+0000 7f60fb412640 1 -- 172.21.15.86:0/1313998131 wait complete. 2024-03-29T11:28:27.251 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:27.261+0000 7f60fb412640 1 librados: shutdown 2024-03-29T11:28:27.264 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:34.072 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:34.198 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.208+0000 7f33a4a4a640 10 monclient: get_monmap_and_config 2024-03-29T11:28:34.198 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.208+0000 7f33a4a4a640 10 monclient: build_initial_monmap 2024-03-29T11:28:34.198 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.208+0000 7f33a4a4a640 10 monclient: monmap: 2024-03-29T11:28:34.199 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:28:34.199 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:34.199 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:28:34.209690+0000 2024-03-29T11:28:34.199 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:28:34.209690+0000 2024-03-29T11:28:34.199 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:34.199 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:34.199 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:28:34.199 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:28:34.199 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:28:34.199 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:34.202 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.209+0000 7f33a4a4a640 1 Processor -- start 2024-03-29T11:28:34.202 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.209+0000 7f33a4a4a640 1 -- start start 2024-03-29T11:28:34.202 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.209+0000 7f33a4a4a640 10 monclient: init 2024-03-29T11:28:34.202 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.210+0000 7f33a4a4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:34.202 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.210+0000 7f33a4a4a640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:28:34.202 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.210+0000 7f33a4a4a640 10 monclient(hunting): picked mon.noname-a con 0x7f33a01422b0 addr v1:172.21.15.86:6789/0 2024-03-29T11:28:34.202 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.210+0000 7f33a4a4a640 10 monclient(hunting): picked mon.noname-c con 0x7f33a0142690 addr v1:172.21.15.196:6789/0 2024-03-29T11:28:34.202 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.210+0000 7f33a4a4a640 10 monclient(hunting): picked mon.noname-b con 0x7f33a00b17d0 addr v1:172.21.15.86:6790/0 2024-03-29T11:28:34.202 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.210+0000 7f33a4a4a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33a0142c80 con 0x7f33a01422b0 2024-03-29T11:28:34.202 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.210+0000 7f33a4a4a640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33a00aae40 con 0x7f33a0142690 2024-03-29T11:28:34.202 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.210+0000 7f33a4a4a640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33a013fbd0 con 0x7f33a00b17d0 2024-03-29T11:28:34.202 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.210+0000 7f33a4a4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:34.203 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.210+0000 7f33a4a4a640 10 monclient(hunting): authenticate will time out at 1338.528730s 2024-03-29T11:28:34.203 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.210+0000 7f339ed9d640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f33a00b17d0 0x7f33a00b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:36280/0 (socket says 172.21.15.86:36280) 2024-03-29T11:28:34.203 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.210+0000 7f339ed9d640 1 -- 172.21.15.86:0/3529977050 learned_addr learned my addr 172.21.15.86:0/3529977050 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:28:34.203 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.211+0000 7f339d59a640 1 -- 172.21.15.86:0/3529977050 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2735734766 0 0) 0x7f33a013fbd0 con 0x7f33a00b17d0 2024-03-29T11:28:34.203 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.211+0000 7f339d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:28:34.203 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.211+0000 7f339d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:34.203 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.211+0000 7f339d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:34.203 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.211+0000 7f339d59a640 10 monclient(hunting): my global_id is 4562 2024-03-29T11:28:34.203 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.211+0000 7f339d59a640 1 -- 172.21.15.86:0/3529977050 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3388003520 con 0x7f33a00b17d0 2024-03-29T11:28:34.203 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.211+0000 7f339d59a640 1 -- 172.21.15.86:0/3529977050 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 551722518 0 0) 0x7f33a00aae40 con 0x7f33a0142690 2024-03-29T11:28:34.203 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.211+0000 7f339d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:28:34.203 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.211+0000 7f339d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:34.203 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.211+0000 7f339d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:34.203 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.211+0000 7f339d59a640 10 monclient(hunting): my global_id is 4465 2024-03-29T11:28:34.204 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.211+0000 7f339d59a640 1 -- 172.21.15.86:0/3529977050 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33a013fbd0 con 0x7f33a0142690 2024-03-29T11:28:34.204 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.211+0000 7f339d59a640 1 -- 172.21.15.86:0/3529977050 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 104352037 0 0) 0x7f3388003520 con 0x7f33a00b17d0 2024-03-29T11:28:34.204 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.211+0000 7f339d59a640 1 -- 172.21.15.86:0/3529977050 >> v1:172.21.15.196:6789/0 conn(0x7f33a0142690 legacy=0x7f33a00b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:34.204 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.211+0000 7f339d59a640 1 -- 172.21.15.86:0/3529977050 >> v1:172.21.15.86:6789/0 conn(0x7f33a01422b0 legacy=0x7f33a013f4c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:34.206 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.213+0000 7f339d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:28:34.206 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.213+0000 7f339d59a640 1 monclient: found mon.noname-b 2024-03-29T11:28:34.206 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.213+0000 7f339d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:34.206 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.213+0000 7f339d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:28:34.206 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.213+0000 7f339d59a640 1 -- 172.21.15.86:0/3529977050 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a0151940 con 0x7f33a00b17d0 2024-03-29T11:28:34.206 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.213+0000 7f339d59a640 10 monclient: _finish_auth 0 2024-03-29T11:28:34.206 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.213+0000 7f339d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:28:34.206 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.213+0000 7f339d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:34.206 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.213+0000 7f339d59a640 1 -- 172.21.15.86:0/3529977050 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f3398001a10 con 0x7f33a00b17d0 2024-03-29T11:28:34.207 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.213+0000 7f339d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:34.207 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.213+0000 7f339d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:34.207 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.213+0000 7f339d59a640 10 monclient: dump: 2024-03-29T11:28:34.207 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:28:34.207 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:34.207 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:34.207 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:34.207 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:34.207 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:34.207 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:28:34.207 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:28:34.207 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:28:34.207 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:34.207 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.213+0000 7f33a4a4a640 5 monclient: authenticate success, global_id 4562 2024-03-29T11:28:34.207 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.213+0000 7f33a4a4a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:34.208 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.213+0000 7f339d59a640 1 -- 172.21.15.86:0/3529977050 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3398001b70 con 0x7f33a00b17d0 2024-03-29T11:28:34.208 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.213+0000 7f339d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:34.208 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.213+0000 7f339d59a640 1 -- 172.21.15.86:0/3529977050 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f3398001f00 con 0x7f33a00b17d0 2024-03-29T11:28:34.208 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.213+0000 7f339d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:34.208 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.213+0000 7f339d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:34.208 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.213+0000 7f339d59a640 10 monclient: dump: 2024-03-29T11:28:34.208 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:28:34.208 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:34.208 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:34.208 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:34.208 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:34.208 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:34.208 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:28:34.208 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:28:34.209 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:28:34.209 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:34.209 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.213+0000 7f33a4a4a640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:34.209 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.213+0000 7f33a4a4a640 4 set_mon_vals no callback set 2024-03-29T11:28:34.209 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.214+0000 7f33a4a4a640 10 monclient: shutdown 2024-03-29T11:28:34.209 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.214+0000 7f33a4a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:34.209 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.214+0000 7f33a4a4a640 1 -- 172.21.15.86:0/3529977050 >> v1:172.21.15.86:6790/0 conn(0x7f33a00b17d0 legacy=0x7f33a00b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:34.209 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.214+0000 7f33a4a4a640 1 -- 172.21.15.86:0/3529977050 shutdown_connections 2024-03-29T11:28:34.209 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.214+0000 7f33a4a4a640 1 -- 172.21.15.86:0/3529977050 >> 172.21.15.86:0/3529977050 conn(0x7f33a0066a00 msgr2=0x7f33a0066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:34.209 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.214+0000 7f33a4a4a640 1 -- 172.21.15.86:0/3529977050 shutdown_connections 2024-03-29T11:28:34.209 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.214+0000 7f33a4a4a640 1 -- 172.21.15.86:0/3529977050 wait complete. 2024-03-29T11:28:34.209 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.216+0000 7f33a4a4a640 10 monclient: build_initial_monmap 2024-03-29T11:28:34.209 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.216+0000 7f33a4a4a640 10 monclient: monmap: 2024-03-29T11:28:34.209 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:28:34.209 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:34.210 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:28:34.210 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:28:34.210 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:34.210 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:34.210 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:28:34.210 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:28:34.210 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:28:34.210 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:34.210 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.216+0000 7f33a4a4a640 1 librados: starting msgr at 2024-03-29T11:28:34.210 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.216+0000 7f33a4a4a640 1 librados: starting objecter 2024-03-29T11:28:34.210 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.216+0000 7f33a4a4a640 1 Processor -- start 2024-03-29T11:28:34.210 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.216+0000 7f33a4a4a640 1 -- start start 2024-03-29T11:28:34.210 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.216+0000 7f33a4a4a640 1 librados: setting wanted keys 2024-03-29T11:28:34.210 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.216+0000 7f33a4a4a640 1 librados: calling monclient init 2024-03-29T11:28:34.210 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.216+0000 7f33a4a4a640 10 monclient: init 2024-03-29T11:28:34.215 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.217+0000 7f33a4a4a640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:34.215 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.217+0000 7f33a4a4a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:28:34.215 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.217+0000 7f33a4a4a640 10 monclient(hunting): picked mon.noname-c con 0x7f33a00b17d0 addr v1:172.21.15.86:6790/0 2024-03-29T11:28:34.215 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.217+0000 7f33a4a4a640 10 monclient(hunting): picked mon.noname-b con 0x7f33a01422b0 addr v1:172.21.15.196:6789/0 2024-03-29T11:28:34.215 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.217+0000 7f33a4a4a640 10 monclient(hunting): picked mon.noname-a con 0x7f33a0142690 addr v1:172.21.15.86:6789/0 2024-03-29T11:28:34.215 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.217+0000 7f33a4a4a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33a01db4a0 con 0x7f33a0142690 2024-03-29T11:28:34.215 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.217+0000 7f33a4a4a640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33a012e700 con 0x7f33a01422b0 2024-03-29T11:28:34.216 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.217+0000 7f33a4a4a640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33a00abca0 con 0x7f33a00b17d0 2024-03-29T11:28:34.216 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.217+0000 7f33a4a4a640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:34.216 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.217+0000 7f33a4a4a640 10 monclient(hunting): authenticate will time out at 1338.535359s 2024-03-29T11:28:34.216 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.217+0000 7f339e59c640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f33a00b17d0 0x7f33a014f270 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:37176/0 (socket says 172.21.15.86:37176) 2024-03-29T11:28:34.216 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.217+0000 7f339e59c640 1 -- 172.21.15.86:0/3543927193 learned_addr learned my addr 172.21.15.86:0/3543927193 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:28:34.216 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.217+0000 7f338effd640 10 client.?.objecter ms_handle_connect 0x7f33a00b17d0 2024-03-29T11:28:34.216 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.217+0000 7f338effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:34.216 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.217+0000 7f338effd640 10 client.?.objecter ms_handle_connect 0x7f33a01422b0 2024-03-29T11:28:34.216 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.217+0000 7f338effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:34.216 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.217+0000 7f338effd640 1 -- 172.21.15.86:0/3543927193 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 225881860 0 0) 0x7f33a00abca0 con 0x7f33a00b17d0 2024-03-29T11:28:34.216 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.217+0000 7f338effd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:34.216 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.217+0000 7f338effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:34.216 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.217+0000 7f338effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:34.217 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.217+0000 7f338effd640 10 monclient(hunting): my global_id is 4565 2024-03-29T11:28:34.217 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.217+0000 7f338effd640 1 -- 172.21.15.86:0/3543927193 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3380004eb0 con 0x7f33a00b17d0 2024-03-29T11:28:34.217 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.217+0000 7f338effd640 1 -- 172.21.15.86:0/3543927193 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1993084594 0 0) 0x7f33a012e700 con 0x7f33a01422b0 2024-03-29T11:28:34.217 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.217+0000 7f338effd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:34.217 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.217+0000 7f338effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:34.217 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.217+0000 7f338effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:34.217 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.217+0000 7f338effd640 10 monclient(hunting): my global_id is 4468 2024-03-29T11:28:34.217 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.217+0000 7f338effd640 1 -- 172.21.15.86:0/3543927193 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f33a00abca0 con 0x7f33a01422b0 2024-03-29T11:28:34.217 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.217+0000 7f338effd640 1 -- 172.21.15.86:0/3543927193 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4034670412 0 0) 0x7f3380004eb0 con 0x7f33a00b17d0 2024-03-29T11:28:34.217 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.217+0000 7f338effd640 1 -- 172.21.15.86:0/3543927193 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f33a012e700 con 0x7f33a00b17d0 2024-03-29T11:28:34.217 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.218+0000 7f338effd640 1 -- 172.21.15.86:0/3543927193 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f3390001960 con 0x7f33a00b17d0 2024-03-29T11:28:34.217 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.218+0000 7f338effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:34.217 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.218+0000 7f338effd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:34.217 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.218+0000 7f338effd640 10 monclient(hunting): dump: 2024-03-29T11:28:34.217 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:28:34.218 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:34.218 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:34.218 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:34.218 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:34.218 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:34.218 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:28:34.218 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:28:34.218 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:28:34.218 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:34.218 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.218+0000 7f338effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:34.218 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.218+0000 7f338effd640 1 -- 172.21.15.86:0/3543927193 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 4098562163 0 0) 0x7f33a00abca0 con 0x7f33a01422b0 2024-03-29T11:28:34.218 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.218+0000 7f338effd640 1 -- 172.21.15.86:0/3543927193 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3380004eb0 con 0x7f33a01422b0 2024-03-29T11:28:34.218 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.218+0000 7f338effd640 1 -- 172.21.15.86:0/3543927193 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3669575989 0 0) 0x7f33a012e700 con 0x7f33a00b17d0 2024-03-29T11:28:34.218 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.218+0000 7f338effd640 1 -- 172.21.15.86:0/3543927193 >> v1:172.21.15.196:6789/0 conn(0x7f33a01422b0 legacy=0x7f33a012d8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:34.219 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.218+0000 7f338effd640 1 -- 172.21.15.86:0/3543927193 >> v1:172.21.15.86:6789/0 conn(0x7f33a0142690 legacy=0x7f33a012dff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:34.219 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.220+0000 7f338effd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:34.219 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.220+0000 7f338effd640 1 monclient: found mon.c 2024-03-29T11:28:34.219 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.220+0000 7f338effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:34.219 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.220+0000 7f338effd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:28:34.219 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.220+0000 7f338effd640 1 -- 172.21.15.86:0/3543927193 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33a00abf80 con 0x7f33a00b17d0 2024-03-29T11:28:34.219 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.220+0000 7f338effd640 10 monclient: _finish_auth 0 2024-03-29T11:28:34.219 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.220+0000 7f338effd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:34.219 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.220+0000 7f338effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:34.219 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.220+0000 7f33a4a4a640 5 monclient: authenticate success, global_id 4565 2024-03-29T11:28:34.219 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.220+0000 7f33a4a4a640 10 monclient: _renew_subs 2024-03-29T11:28:34.219 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.220+0000 7f33a4a4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:28:34.219 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.220+0000 7f33a4a4a640 1 -- 172.21.15.86:0/3543927193 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33a00ac260 con 0x7f33a00b17d0 2024-03-29T11:28:34.219 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.220+0000 7f33a4a4a640 10 client.4565.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:34.219 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.220+0000 7f33a4a4a640 10 monclient: _renew_subs 2024-03-29T11:28:34.219 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.220+0000 7f33a4a4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:28:34.220 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.220+0000 7f33a4a4a640 1 -- 172.21.15.86:0/3543927193 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f33a01db030 con 0x7f33a00b17d0 2024-03-29T11:28:34.220 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.220+0000 7f33a4a4a640 1 librados: init done 2024-03-29T11:28:34.220 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.220+0000 7f338effd640 1 -- 172.21.15.86:0/3543927193 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3390001ac0 con 0x7f33a00b17d0 2024-03-29T11:28:34.220 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.220+0000 7f338effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:34.220 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.220+0000 7f338effd640 1 -- 172.21.15.86:0/3543927193 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f33900038a0 con 0x7f33a00b17d0 2024-03-29T11:28:34.220 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.220+0000 7f338effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:34.220 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.220+0000 7f338effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:34.220 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.220+0000 7f338effd640 10 monclient: dump: 2024-03-29T11:28:34.220 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:28:34.220 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:34.220 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:34.220 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:34.220 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:34.220 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:34.221 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:28:34.221 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:28:34.221 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:28:34.221 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:34.221 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.220+0000 7f339cd99640 4 set_mon_vals no callback set 2024-03-29T11:28:34.221 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.221+0000 7f338effd640 1 -- 172.21.15.86:0/3543927193 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f3390003a00 con 0x7f33a00b17d0 2024-03-29T11:28:34.221 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.221+0000 7f33a4a4a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:34.221 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.221+0000 7f33a4a4a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:34.221 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.221+0000 7f33a4a4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:28:34.221 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.221+0000 7f33a4a4a640 1 -- 172.21.15.86:0/3543927193 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3384005080 con 0x7f33a00b17d0 2024-03-29T11:28:34.221 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.222+0000 7f338effd640 1 -- 172.21.15.86:0/3543927193 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f3390084a10 con 0x7f33a00b17d0 2024-03-29T11:28:34.221 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.222+0000 7f338effd640 10 client.4565.objecter ms_dispatch 0x7f33a0000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:34.221 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.222+0000 7f338effd640 3 client.4565.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:34.221 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.222+0000 7f338effd640 3 client.4565.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:34.221 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.222+0000 7f338effd640 20 client.4565.objecter dump_active .. 0 homeless 2024-03-29T11:28:34.222 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.222+0000 7f339dd9b640 10 monclient: get_auth_request con 0x7f338006ddd0 auth_method 0 2024-03-29T11:28:34.222 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.222+0000 7f338effd640 10 client.4565.objecter ms_handle_connect 0x7f338006ddd0 2024-03-29T11:28:34.222 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.228+0000 7f338effd640 1 -- 172.21.15.86:0/3543927193 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f33900502f0 con 0x7f33a00b17d0 2024-03-29T11:28:34.222 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.228+0000 7f338effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:34.222 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.228+0000 7f338effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:34.492 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.502+0000 7f33a4a4a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:34.492 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.502+0000 7f33a4a4a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:34.492 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.502+0000 7f33a4a4a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:28:34.492 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.502+0000 7f33a4a4a640 1 -- 172.21.15.86:0/3543927193 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3384005330 con 0x7f33a00b17d0 2024-03-29T11:28:34.493 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.502+0000 7f338effd640 1 -- 172.21.15.86:0/3543927193 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f3390052220 con 0x7f33a00b17d0 2024-03-29T11:28:34.493 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.502+0000 7f338effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:34.493 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.502+0000 7f338effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:34.493 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:28:34.493 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:34.495 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.505+0000 7f33a4a4a640 10 librados: watch_flush enter 2024-03-29T11:28:34.495 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.505+0000 7f33a4a4a640 10 librados: watch_flush exit 2024-03-29T11:28:34.495 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.505+0000 7f33a4a4a640 20 client.4565.objecter shutdown clearing up homeless session... 2024-03-29T11:28:34.496 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.505+0000 7f33a4a4a640 10 client.4565.objecter successfully canceled tick 2024-03-29T11:28:34.496 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.505+0000 7f33a4a4a640 1 -- 172.21.15.86:0/3543927193 >> v1:172.21.15.196:6816/588110318 conn(0x7f338006ddd0 legacy=0x7f3380070290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:34.496 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.505+0000 7f33a4a4a640 10 monclient: shutdown 2024-03-29T11:28:34.496 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.505+0000 7f33a4a4a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:34.496 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.505+0000 7f33a4a4a640 1 -- 172.21.15.86:0/3543927193 >> v1:172.21.15.86:6790/0 conn(0x7f33a00b17d0 legacy=0x7f33a014f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:34.496 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.505+0000 7f33a4a4a640 1 -- 172.21.15.86:0/3543927193 shutdown_connections 2024-03-29T11:28:34.496 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.505+0000 7f33a4a4a640 1 -- 172.21.15.86:0/3543927193 >> 172.21.15.86:0/3543927193 conn(0x7f33a0066a00 msgr2=0x7f33a013f290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:34.496 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.505+0000 7f33a4a4a640 1 -- 172.21.15.86:0/3543927193 shutdown_connections 2024-03-29T11:28:34.496 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.506+0000 7f33a4a4a640 1 -- 172.21.15.86:0/3543927193 wait complete. 2024-03-29T11:28:34.496 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:34.506+0000 7f33a4a4a640 1 librados: shutdown 2024-03-29T11:28:34.509 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:41.317 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:41.440 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.450+0000 7f807a9e6640 10 monclient: get_monmap_and_config 2024-03-29T11:28:41.441 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.450+0000 7f807a9e6640 10 monclient: build_initial_monmap 2024-03-29T11:28:41.441 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.450+0000 7f807a9e6640 10 monclient: monmap: 2024-03-29T11:28:41.441 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:28:41.441 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:41.441 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:28:41.451617+0000 2024-03-29T11:28:41.441 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:28:41.451617+0000 2024-03-29T11:28:41.441 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:41.441 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:41.441 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:28:41.441 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:28:41.441 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:28:41.441 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:41.443 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.451+0000 7f807a9e6640 1 Processor -- start 2024-03-29T11:28:41.443 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.451+0000 7f807a9e6640 1 -- start start 2024-03-29T11:28:41.443 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.451+0000 7f807a9e6640 10 monclient: init 2024-03-29T11:28:41.443 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.452+0000 7f807a9e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:41.443 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.452+0000 7f807a9e6640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:28:41.443 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.452+0000 7f807a9e6640 10 monclient(hunting): picked mon.noname-c con 0x7f80740b0500 addr v1:172.21.15.196:6789/0 2024-03-29T11:28:41.443 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.452+0000 7f807a9e6640 10 monclient(hunting): picked mon.noname-b con 0x7f80740b08e0 addr v1:172.21.15.86:6790/0 2024-03-29T11:28:41.443 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.452+0000 7f80737fe640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f80740b08e0 0x7f807413d3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:38244/0 (socket says 172.21.15.86:38244) 2024-03-29T11:28:41.443 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.453+0000 7f8073fff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f80740b0500 0x7f807413ff10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:56878/0 (socket says 172.21.15.86:56878) 2024-03-29T11:28:41.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.453+0000 7f807a9e6640 10 monclient(hunting): picked mon.noname-a con 0x7f80740aeca0 addr v1:172.21.15.86:6789/0 2024-03-29T11:28:41.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.453+0000 7f80737fe640 1 -- 172.21.15.86:0/2390655110 learned_addr learned my addr 172.21.15.86:0/2390655110 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:28:41.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.453+0000 7f807a9e6640 1 -- 172.21.15.86:0/2390655110 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8074140620 con 0x7f80740aeca0 2024-03-29T11:28:41.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.453+0000 7f807a9e6640 1 -- 172.21.15.86:0/2390655110 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f807413daf0 con 0x7f80740b0500 2024-03-29T11:28:41.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.453+0000 7f807a9e6640 1 -- 172.21.15.86:0/2390655110 --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80740ae180 con 0x7f80740b08e0 2024-03-29T11:28:41.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.453+0000 7f807a9e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:41.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.453+0000 7f807a9e6640 10 monclient(hunting): authenticate will time out at 1345.771234s 2024-03-29T11:28:41.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.453+0000 7f8072ffd640 1 -- 172.21.15.86:0/2390655110 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 237738203 0 0) 0x7f80740ae180 con 0x7f80740b08e0 2024-03-29T11:28:41.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.453+0000 7f8072ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:28:41.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.453+0000 7f8072ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:41.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.453+0000 7f8072ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:41.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.453+0000 7f8072ffd640 10 monclient(hunting): my global_id is 4568 2024-03-29T11:28:41.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.453+0000 7f8072ffd640 1 -- 172.21.15.86:0/2390655110 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f805c003520 con 0x7f80740b08e0 2024-03-29T11:28:41.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.453+0000 7f8072ffd640 1 -- 172.21.15.86:0/2390655110 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2869770029 0 0) 0x7f805c003520 con 0x7f80740b08e0 2024-03-29T11:28:41.444 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.453+0000 7f8072ffd640 1 -- 172.21.15.86:0/2390655110 >> v1:172.21.15.196:6789/0 conn(0x7f80740b0500 legacy=0x7f807413ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:28:41.445 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.453+0000 7f8072ffd640 1 -- 172.21.15.86:0/2390655110 >> v1:172.21.15.86:6789/0 conn(0x7f80740aeca0 legacy=0x7f807414b7d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:41.448 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.454+0000 7f8072ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:28:41.448 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.454+0000 7f8072ffd640 1 monclient: found mon.noname-b 2024-03-29T11:28:41.448 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.454+0000 7f8072ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:41.448 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.454+0000 7f8072ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:28:41.448 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.454+0000 7f8072ffd640 1 -- 172.21.15.86:0/2390655110 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80740accf0 con 0x7f80740b08e0 2024-03-29T11:28:41.448 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.454+0000 7f8072ffd640 10 monclient: _finish_auth 0 2024-03-29T11:28:41.448 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.454+0000 7f8072ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:28:41.448 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.454+0000 7f8072ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:41.448 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.454+0000 7f8072ffd640 1 -- 172.21.15.86:0/2390655110 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f8068001690 con 0x7f80740b08e0 2024-03-29T11:28:41.448 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.454+0000 7f8072ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:41.448 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.454+0000 7f8072ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:41.449 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.454+0000 7f8072ffd640 10 monclient: dump: 2024-03-29T11:28:41.449 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:28:41.449 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:41.449 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:41.449 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:41.449 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:41.449 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:41.449 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:28:41.450 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:28:41.450 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:28:41.450 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:41.450 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.455+0000 7f807a9e6640 5 monclient: authenticate success, global_id 4568 2024-03-29T11:28:41.450 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.455+0000 7f807a9e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:41.450 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.455+0000 7f8072ffd640 1 -- 172.21.15.86:0/2390655110 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8068001860 con 0x7f80740b08e0 2024-03-29T11:28:41.450 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.455+0000 7f8072ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:41.450 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.455+0000 7f8072ffd640 1 -- 172.21.15.86:0/2390655110 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f8068001cb0 con 0x7f80740b08e0 2024-03-29T11:28:41.450 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.455+0000 7f8072ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:41.450 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.455+0000 7f8072ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:41.450 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.455+0000 7f8072ffd640 10 monclient: dump: 2024-03-29T11:28:41.450 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:28:41.450 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:41.450 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:41.450 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:41.451 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:41.451 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:41.451 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:28:41.451 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:28:41.451 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:28:41.451 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:41.451 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.455+0000 7f807a9e6640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:41.451 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.455+0000 7f807a9e6640 4 set_mon_vals no callback set 2024-03-29T11:28:41.451 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.456+0000 7f807a9e6640 10 monclient: shutdown 2024-03-29T11:28:41.451 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.456+0000 7f807a9e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:41.451 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.456+0000 7f807a9e6640 1 -- 172.21.15.86:0/2390655110 >> v1:172.21.15.86:6790/0 conn(0x7f80740b08e0 legacy=0x7f807413d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:41.452 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.456+0000 7f807a9e6640 1 -- 172.21.15.86:0/2390655110 shutdown_connections 2024-03-29T11:28:41.452 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.456+0000 7f807a9e6640 1 -- 172.21.15.86:0/2390655110 >> 172.21.15.86:0/2390655110 conn(0x7f807406b660 msgr2=0x7f807406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:41.452 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.456+0000 7f807a9e6640 1 -- 172.21.15.86:0/2390655110 shutdown_connections 2024-03-29T11:28:41.452 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.456+0000 7f807a9e6640 1 -- 172.21.15.86:0/2390655110 wait complete. 2024-03-29T11:28:41.452 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.457+0000 7f807a9e6640 10 monclient: build_initial_monmap 2024-03-29T11:28:41.452 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.457+0000 7f807a9e6640 10 monclient: monmap: 2024-03-29T11:28:41.452 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:28:41.452 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:41.452 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:28:41.452 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:28:41.452 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:41.453 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:41.453 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:28:41.453 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:28:41.453 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:28:41.453 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:41.453 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.457+0000 7f807a9e6640 1 librados: starting msgr at 2024-03-29T11:28:41.453 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.457+0000 7f807a9e6640 1 librados: starting objecter 2024-03-29T11:28:41.453 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.457+0000 7f807a9e6640 1 Processor -- start 2024-03-29T11:28:41.453 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.457+0000 7f807a9e6640 1 -- start start 2024-03-29T11:28:41.453 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.457+0000 7f807a9e6640 1 librados: setting wanted keys 2024-03-29T11:28:41.453 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.457+0000 7f807a9e6640 1 librados: calling monclient init 2024-03-29T11:28:41.453 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.457+0000 7f807a9e6640 10 monclient: init 2024-03-29T11:28:41.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.458+0000 7f807a9e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:41.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.458+0000 7f807a9e6640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:28:41.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.458+0000 7f807a9e6640 10 monclient(hunting): picked mon.noname-a con 0x7f80740aeca0 addr v1:172.21.15.86:6789/0 2024-03-29T11:28:41.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.458+0000 7f807a9e6640 10 monclient(hunting): picked mon.noname-b con 0x7f80740b0500 addr v1:172.21.15.196:6789/0 2024-03-29T11:28:41.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.458+0000 7f807a9e6640 10 monclient(hunting): picked mon.noname-c con 0x7f80741d7130 addr v1:172.21.15.86:6790/0 2024-03-29T11:28:41.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.458+0000 7f807a9e6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f807406fc70 con 0x7f80740aeca0 2024-03-29T11:28:41.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.458+0000 7f807a9e6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80741da850 con 0x7f80740b0500 2024-03-29T11:28:41.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.459+0000 7f807a9e6640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80741db480 con 0x7f80741d7130 2024-03-29T11:28:41.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.459+0000 7f80737fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f80740b0500 0x7f807413b790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:57306/0 (socket says 172.21.15.86:57306) 2024-03-29T11:28:41.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.459+0000 7f80737fe640 1 -- 172.21.15.86:0/3368278638 learned_addr learned my addr 172.21.15.86:0/3368278638 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:28:41.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.459+0000 7f8070ff9640 10 client.?.objecter ms_handle_connect 0x7f80740b0500 2024-03-29T11:28:41.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.459+0000 7f8070ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:41.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.459+0000 7f807a9e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:41.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.459+0000 7f807a9e6640 10 monclient(hunting): authenticate will time out at 1345.777244s 2024-03-29T11:28:41.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.459+0000 7f8070ff9640 10 client.?.objecter ms_handle_connect 0x7f80741d7130 2024-03-29T11:28:41.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.459+0000 7f8070ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:41.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.459+0000 7f8070ff9640 1 -- 172.21.15.86:0/3368278638 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 941932674 0 0) 0x7f80741da850 con 0x7f80740b0500 2024-03-29T11:28:41.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.459+0000 7f8070ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:28:41.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.459+0000 7f8070ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:41.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.459+0000 7f8070ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:41.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.459+0000 7f8070ff9640 10 monclient(hunting): my global_id is 4471 2024-03-29T11:28:41.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.459+0000 7f8070ff9640 1 -- 172.21.15.86:0/3368278638 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8060004eb0 con 0x7f80740b0500 2024-03-29T11:28:41.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.460+0000 7f8070ff9640 1 -- 172.21.15.86:0/3368278638 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2253334114 0 0) 0x7f80741db480 con 0x7f80741d7130 2024-03-29T11:28:41.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.460+0000 7f8070ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:28:41.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.460+0000 7f8070ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:41.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.460+0000 7f8070ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:41.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.460+0000 7f8070ff9640 10 monclient(hunting): my global_id is 4571 2024-03-29T11:28:41.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.460+0000 7f8070ff9640 1 -- 172.21.15.86:0/3368278638 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80741da850 con 0x7f80741d7130 2024-03-29T11:28:41.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.460+0000 7f8070ff9640 1 -- 172.21.15.86:0/3368278638 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1791003985 0 0) 0x7f80741da850 con 0x7f80741d7130 2024-03-29T11:28:41.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.460+0000 7f8070ff9640 1 -- 172.21.15.86:0/3368278638 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80741db480 con 0x7f80741d7130 2024-03-29T11:28:41.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.460+0000 7f8070ff9640 1 -- 172.21.15.86:0/3368278638 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f806c001b30 con 0x7f80741d7130 2024-03-29T11:28:41.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.460+0000 7f8070ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:41.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.460+0000 7f8070ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:41.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.460+0000 7f8070ff9640 10 monclient(hunting): dump: 2024-03-29T11:28:41.461 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:28:41.461 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:41.461 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:41.461 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:41.461 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:41.461 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:41.461 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:28:41.461 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:28:41.461 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:28:41.462 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:41.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.460+0000 7f8070ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:41.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.461+0000 7f8070ff9640 1 -- 172.21.15.86:0/3368278638 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2130680465 0 0) 0x7f80741db480 con 0x7f80741d7130 2024-03-29T11:28:41.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.461+0000 7f8070ff9640 1 -- 172.21.15.86:0/3368278638 >> v1:172.21.15.196:6789/0 conn(0x7f80740b0500 legacy=0x7f807413b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:41.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.461+0000 7f8070ff9640 1 -- 172.21.15.86:0/3368278638 >> v1:172.21.15.86:6789/0 conn(0x7f80740aeca0 legacy=0x7f80741dad70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:41.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.461+0000 7f8070ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:28:41.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.461+0000 7f8070ff9640 1 monclient: found mon.c 2024-03-29T11:28:41.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.461+0000 7f8070ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:41.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.461+0000 7f8070ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:28:41.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.461+0000 7f8070ff9640 1 -- 172.21.15.86:0/3368278638 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80740ac810 con 0x7f80741d7130 2024-03-29T11:28:41.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.461+0000 7f8070ff9640 10 monclient: _finish_auth 0 2024-03-29T11:28:41.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.461+0000 7f8070ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:28:41.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.461+0000 7f8070ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:41.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.461+0000 7f807a9e6640 5 monclient: authenticate success, global_id 4571 2024-03-29T11:28:41.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.461+0000 7f807a9e6640 10 monclient: _renew_subs 2024-03-29T11:28:41.463 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.461+0000 7f807a9e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:28:41.463 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.461+0000 7f807a9e6640 1 -- 172.21.15.86:0/3368278638 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80740acac0 con 0x7f80741d7130 2024-03-29T11:28:41.463 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.461+0000 7f807a9e6640 10 client.4571.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:41.463 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.462+0000 7f807a9e6640 10 monclient: _renew_subs 2024-03-29T11:28:41.463 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.462+0000 7f807a9e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:28:41.463 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.462+0000 7f8070ff9640 1 -- 172.21.15.86:0/3368278638 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f806c002040 con 0x7f80741d7130 2024-03-29T11:28:41.463 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.462+0000 7f807a9e6640 1 -- 172.21.15.86:0/3368278638 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f807413c080 con 0x7f80741d7130 2024-03-29T11:28:41.463 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.462+0000 7f8070ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:41.463 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.462+0000 7f8070ff9640 1 -- 172.21.15.86:0/3368278638 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f806c003b10 con 0x7f80741d7130 2024-03-29T11:28:41.463 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.462+0000 7f8070ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:41.463 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.462+0000 7f8070ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:41.463 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.462+0000 7f8070ff9640 10 monclient: dump: 2024-03-29T11:28:41.463 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:28:41.463 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:41.463 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:41.464 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:41.464 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:41.464 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:41.464 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:28:41.464 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:28:41.464 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:28:41.464 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:41.464 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.462+0000 7f807a9e6640 1 librados: init done 2024-03-29T11:28:41.464 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.462+0000 7f80727fc640 4 set_mon_vals no callback set 2024-03-29T11:28:41.464 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.463+0000 7f807a9e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:41.464 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.463+0000 7f807a9e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:41.464 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.463+0000 7f807a9e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:28:41.464 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.463+0000 7f807a9e6640 1 -- 172.21.15.86:0/3368278638 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80740ad0c0 con 0x7f80741d7130 2024-03-29T11:28:41.464 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.464+0000 7f8070ff9640 1 -- 172.21.15.86:0/3368278638 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f806c019ae0 con 0x7f80741d7130 2024-03-29T11:28:41.464 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.465+0000 7f8070ff9640 1 -- 172.21.15.86:0/3368278638 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f806c083ab0 con 0x7f80741d7130 2024-03-29T11:28:41.465 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.465+0000 7f8070ff9640 10 client.4571.objecter ms_dispatch 0x7f8074000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:41.465 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.465+0000 7f8070ff9640 3 client.4571.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:41.465 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.465+0000 7f8070ff9640 3 client.4571.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:41.465 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.465+0000 7f8073fff640 10 monclient: get_auth_request con 0x7f806006dd60 auth_method 0 2024-03-29T11:28:41.465 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.465+0000 7f8070ff9640 20 client.4571.objecter dump_active .. 0 homeless 2024-03-29T11:28:41.465 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.465+0000 7f8070ff9640 10 client.4571.objecter ms_handle_connect 0x7f806006dd60 2024-03-29T11:28:41.465 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.471+0000 7f8070ff9640 1 -- 172.21.15.86:0/3368278638 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f806c04f390 con 0x7f80741d7130 2024-03-29T11:28:41.465 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.471+0000 7f8070ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:41.465 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.471+0000 7f8070ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:41.733 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.742+0000 7f807a9e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:41.733 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.742+0000 7f807a9e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:41.733 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.742+0000 7f807a9e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:28:41.733 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.742+0000 7f807a9e6640 1 -- 172.21.15.86:0/3368278638 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8074075f10 con 0x7f80741d7130 2024-03-29T11:28:41.733 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.742+0000 7f8070ff9640 1 -- 172.21.15.86:0/3368278638 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f806c0512c0 con 0x7f80741d7130 2024-03-29T11:28:41.733 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.743+0000 7f8070ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:41.733 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.743+0000 7f8070ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:41.734 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:28:41.734 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:41.738 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.747+0000 7f807a9e6640 10 librados: watch_flush enter 2024-03-29T11:28:41.738 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.747+0000 7f807a9e6640 10 librados: watch_flush exit 2024-03-29T11:28:41.738 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.747+0000 7f807a9e6640 20 client.4571.objecter shutdown clearing up homeless session... 2024-03-29T11:28:41.738 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.747+0000 7f807a9e6640 10 client.4571.objecter successfully canceled tick 2024-03-29T11:28:41.738 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.748+0000 7f807a9e6640 1 -- 172.21.15.86:0/3368278638 >> v1:172.21.15.196:6816/588110318 conn(0x7f806006dd60 legacy=0x7f8060070220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:41.738 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.748+0000 7f807a9e6640 10 monclient: shutdown 2024-03-29T11:28:41.739 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.748+0000 7f807a9e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:41.739 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.748+0000 7f807a9e6640 1 -- 172.21.15.86:0/3368278638 >> v1:172.21.15.86:6790/0 conn(0x7f80741d7130 legacy=0x7f80741e2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:41.739 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.748+0000 7f807a9e6640 1 -- 172.21.15.86:0/3368278638 shutdown_connections 2024-03-29T11:28:41.739 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.748+0000 7f807a9e6640 1 -- 172.21.15.86:0/3368278638 >> 172.21.15.86:0/3368278638 conn(0x7f807406b660 msgr2=0x7f80740b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:41.739 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.748+0000 7f807a9e6640 1 -- 172.21.15.86:0/3368278638 shutdown_connections 2024-03-29T11:28:41.739 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.748+0000 7f807a9e6640 1 -- 172.21.15.86:0/3368278638 wait complete. 2024-03-29T11:28:41.739 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:41.748+0000 7f807a9e6640 1 librados: shutdown 2024-03-29T11:28:41.752 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:48.560 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:48.697 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.707+0000 7fcc871d7640 10 monclient: get_monmap_and_config 2024-03-29T11:28:48.697 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.707+0000 7fcc871d7640 10 monclient: build_initial_monmap 2024-03-29T11:28:48.697 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.707+0000 7fcc871d7640 10 monclient: monmap: 2024-03-29T11:28:48.698 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:28:48.698 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:48.698 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:28:48.708152+0000 2024-03-29T11:28:48.698 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:28:48.708152+0000 2024-03-29T11:28:48.698 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:48.698 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:48.698 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:28:48.698 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:28:48.698 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:28:48.698 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:48.699 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.708+0000 7fcc871d7640 1 Processor -- start 2024-03-29T11:28:48.699 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.708+0000 7fcc871d7640 1 -- start start 2024-03-29T11:28:48.699 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.708+0000 7fcc871d7640 10 monclient: init 2024-03-29T11:28:48.700 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.709+0000 7fcc871d7640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:48.700 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.709+0000 7fcc871d7640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:28:48.700 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.709+0000 7fcc871d7640 10 monclient(hunting): picked mon.noname-c con 0x7fcc800b0770 addr v1:172.21.15.196:6789/0 2024-03-29T11:28:48.700 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.709+0000 7fcc871d7640 10 monclient(hunting): picked mon.noname-a con 0x7fcc800ad850 addr v1:172.21.15.86:6789/0 2024-03-29T11:28:48.700 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.709+0000 7fcc871d7640 10 monclient(hunting): picked mon.noname-b con 0x7fcc8013c1b0 addr v1:172.21.15.86:6790/0 2024-03-29T11:28:48.700 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.709+0000 7fcc871d7640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc800ac8e0 con 0x7fcc800ad850 2024-03-29T11:28:48.701 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.709+0000 7fcc871d7640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc800ad330 con 0x7fcc800b0770 2024-03-29T11:28:48.701 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.709+0000 7fcc871d7640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc800acfe0 con 0x7fcc8013c1b0 2024-03-29T11:28:48.701 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.709+0000 7fcc871d7640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:48.701 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.709+0000 7fcc871d7640 10 monclient(hunting): authenticate will time out at 1353.027579s 2024-03-29T11:28:48.701 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.710+0000 7fcc85774640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7fcc8013c1b0 0x7fcc8014a760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:42878/0 (socket says 172.21.15.86:42878) 2024-03-29T11:28:48.704 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.712+0000 7fcc85774640 1 -- 172.21.15.86:0/2340761947 learned_addr learned my addr 172.21.15.86:0/2340761947 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:28:48.704 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.712+0000 7fcc7f7fe640 1 -- 172.21.15.86:0/2340761947 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3110036117 0 0) 0x7fcc800ac8e0 con 0x7fcc800ad850 2024-03-29T11:28:48.704 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.712+0000 7fcc7f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:48.704 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.712+0000 7fcc7f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:48.704 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.712+0000 7fcc7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:48.704 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.712+0000 7fcc7f7fe640 10 monclient(hunting): my global_id is 4494 2024-03-29T11:28:48.704 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.712+0000 7fcc7f7fe640 1 -- 172.21.15.86:0/2340761947 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc70003520 con 0x7fcc800ad850 2024-03-29T11:28:48.704 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.712+0000 7fcc7f7fe640 1 -- 172.21.15.86:0/2340761947 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2803084948 0 0) 0x7fcc800acfe0 con 0x7fcc8013c1b0 2024-03-29T11:28:48.704 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.712+0000 7fcc7f7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:28:48.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.712+0000 7fcc7f7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:48.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.712+0000 7fcc7f7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:48.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.712+0000 7fcc7f7fe640 10 monclient(hunting): my global_id is 4574 2024-03-29T11:28:48.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.712+0000 7fcc7f7fe640 1 -- 172.21.15.86:0/2340761947 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc700048a0 con 0x7fcc8013c1b0 2024-03-29T11:28:48.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.713+0000 7fcc7f7fe640 1 -- 172.21.15.86:0/2340761947 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3060840366 0 0) 0x7fcc70003520 con 0x7fcc800ad850 2024-03-29T11:28:48.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.713+0000 7fcc7f7fe640 1 -- 172.21.15.86:0/2340761947 >> v1:172.21.15.86:6790/0 conn(0x7fcc8013c1b0 legacy=0x7fcc8014a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:48.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.713+0000 7fcc7f7fe640 1 -- 172.21.15.86:0/2340761947 >> v1:172.21.15.196:6789/0 conn(0x7fcc800b0770 legacy=0x7fcc80072070 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:48.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.713+0000 7fcc7f7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:28:48.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.713+0000 7fcc7f7fe640 1 monclient: found mon.noname-a 2024-03-29T11:28:48.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.713+0000 7fcc7f7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:48.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.713+0000 7fcc7f7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.86:6789/0 2024-03-29T11:28:48.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.713+0000 7fcc7f7fe640 1 -- 172.21.15.86:0/2340761947 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc800ab1f0 con 0x7fcc800ad850 2024-03-29T11:28:48.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.713+0000 7fcc7f7fe640 10 monclient: _finish_auth 0 2024-03-29T11:28:48.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.713+0000 7fcc7f7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:28:48.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.713+0000 7fcc7f7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:48.706 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.713+0000 7fcc871d7640 5 monclient: authenticate success, global_id 4494 2024-03-29T11:28:48.706 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.713+0000 7fcc871d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:48.706 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.713+0000 7fcc7f7fe640 1 -- 172.21.15.86:0/2340761947 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fcc6c001670 con 0x7fcc800ad850 2024-03-29T11:28:48.706 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.713+0000 7fcc7f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:48.706 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.713+0000 7fcc7f7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:28:48.706 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.713+0000 7fcc7f7fe640 10 monclient: dump: 2024-03-29T11:28:48.706 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:28:48.706 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:48.706 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:48.706 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:48.706 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:48.706 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:48.706 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:28:48.706 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:28:48.707 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:28:48.707 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:48.707 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.713+0000 7fcc871d7640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:48.707 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.713+0000 7fcc7f7fe640 1 -- 172.21.15.86:0/2340761947 <== mon.0 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc6c001850 con 0x7fcc800ad850 2024-03-29T11:28:48.707 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.713+0000 7fcc7f7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:48.707 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.713+0000 7fcc7f7fe640 1 -- 172.21.15.86:0/2340761947 <== mon.0 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fcc6c001c60 con 0x7fcc800ad850 2024-03-29T11:28:48.707 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.713+0000 7fcc7f7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:48.707 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.713+0000 7fcc7f7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:28:48.707 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.713+0000 7fcc7f7fe640 10 monclient: dump: 2024-03-29T11:28:48.707 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:28:48.707 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:48.707 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:48.707 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:48.707 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:48.707 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:48.708 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:28:48.708 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:28:48.708 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:28:48.708 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:48.708 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.714+0000 7fcc871d7640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:48.708 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.714+0000 7fcc871d7640 4 set_mon_vals no callback set 2024-03-29T11:28:48.710 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.714+0000 7fcc871d7640 10 monclient: shutdown 2024-03-29T11:28:48.710 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.714+0000 7fcc871d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:48.710 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.714+0000 7fcc871d7640 1 -- 172.21.15.86:0/2340761947 >> v1:172.21.15.86:6789/0 conn(0x7fcc800ad850 legacy=0x7fcc800aaae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:48.710 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.714+0000 7fcc871d7640 1 -- 172.21.15.86:0/2340761947 shutdown_connections 2024-03-29T11:28:48.710 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.714+0000 7fcc871d7640 1 -- 172.21.15.86:0/2340761947 >> 172.21.15.86:0/2340761947 conn(0x7fcc8006b660 msgr2=0x7fcc8006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:48.711 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.715+0000 7fcc871d7640 1 -- 172.21.15.86:0/2340761947 shutdown_connections 2024-03-29T11:28:48.711 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.715+0000 7fcc871d7640 1 -- 172.21.15.86:0/2340761947 wait complete. 2024-03-29T11:28:48.711 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.716+0000 7fcc871d7640 10 monclient: build_initial_monmap 2024-03-29T11:28:48.711 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.716+0000 7fcc871d7640 10 monclient: monmap: 2024-03-29T11:28:48.711 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:28:48.711 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:48.711 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:28:48.711 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:28:48.711 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:48.711 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:48.711 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:28:48.711 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:28:48.711 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:28:48.711 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:48.712 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.717+0000 7fcc871d7640 1 librados: starting msgr at 2024-03-29T11:28:48.712 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.717+0000 7fcc871d7640 1 librados: starting objecter 2024-03-29T11:28:48.712 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.717+0000 7fcc871d7640 1 Processor -- start 2024-03-29T11:28:48.712 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.717+0000 7fcc871d7640 1 -- start start 2024-03-29T11:28:48.712 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.717+0000 7fcc871d7640 1 librados: setting wanted keys 2024-03-29T11:28:48.713 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.717+0000 7fcc871d7640 1 librados: calling monclient init 2024-03-29T11:28:48.713 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.717+0000 7fcc871d7640 10 monclient: init 2024-03-29T11:28:48.713 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.718+0000 7fcc871d7640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:48.713 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.718+0000 7fcc871d7640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:28:48.713 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.718+0000 7fcc871d7640 10 monclient(hunting): picked mon.noname-a con 0x7fcc800ad850 addr v1:172.21.15.86:6789/0 2024-03-29T11:28:48.713 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.718+0000 7fcc871d7640 10 monclient(hunting): picked mon.noname-b con 0x7fcc800b0770 addr v1:172.21.15.196:6789/0 2024-03-29T11:28:48.713 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.718+0000 7fcc871d7640 10 monclient(hunting): picked mon.noname-c con 0x7fcc8013c1b0 addr v1:172.21.15.86:6790/0 2024-03-29T11:28:48.713 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.718+0000 7fcc871d7640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc8013e3c0 con 0x7fcc800ad850 2024-03-29T11:28:48.713 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.718+0000 7fcc871d7640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc8013e1a0 con 0x7fcc800b0770 2024-03-29T11:28:48.713 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.718+0000 7fcc871d7640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcc8013e6a0 con 0x7fcc8013c1b0 2024-03-29T11:28:48.713 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.718+0000 7fcc871d7640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:48.714 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.718+0000 7fcc871d7640 10 monclient(hunting): authenticate will time out at 1353.036473s 2024-03-29T11:28:48.719 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.721+0000 7fcc84f73640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fcc800ad850 0x7fcc8013da90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:38363/0 (socket says 172.21.15.86:38363) 2024-03-29T11:28:48.719 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.721+0000 7fcc84f73640 1 -- 172.21.15.86:0/1741026112 learned_addr learned my addr 172.21.15.86:0/1741026112 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:28:48.719 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.721+0000 7fcc7d7fa640 10 client.?.objecter ms_handle_connect 0x7fcc800ad850 2024-03-29T11:28:48.719 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.721+0000 7fcc7d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:48.719 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.721+0000 7fcc7d7fa640 10 client.?.objecter ms_handle_connect 0x7fcc8013c1b0 2024-03-29T11:28:48.719 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.721+0000 7fcc7d7fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:48.719 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.721+0000 7fcc7d7fa640 1 -- 172.21.15.86:0/1741026112 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3719103071 0 0) 0x7fcc8013e6a0 con 0x7fcc8013c1b0 2024-03-29T11:28:48.719 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.721+0000 7fcc7d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:28:48.719 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.721+0000 7fcc7d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:48.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.721+0000 7fcc7d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:48.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.721+0000 7fcc7d7fa640 10 monclient(hunting): my global_id is 4577 2024-03-29T11:28:48.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.721+0000 7fcc7d7fa640 1 -- 172.21.15.86:0/1741026112 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc64004eb0 con 0x7fcc8013c1b0 2024-03-29T11:28:48.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.721+0000 7fcc7d7fa640 1 -- 172.21.15.86:0/1741026112 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1994296605 0 0) 0x7fcc8013e3c0 con 0x7fcc800ad850 2024-03-29T11:28:48.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.721+0000 7fcc7d7fa640 10 monclient(hunting): _negotiate 2024-03-29T11:28:48.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.721+0000 7fcc7d7fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:48.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.721+0000 7fcc7d7fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:48.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.721+0000 7fcc7d7fa640 10 monclient(hunting): my global_id is 4497 2024-03-29T11:28:48.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.721+0000 7fcc7d7fa640 1 -- 172.21.15.86:0/1741026112 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcc8013e6a0 con 0x7fcc800ad850 2024-03-29T11:28:48.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.721+0000 7fcc7d7fa640 1 -- 172.21.15.86:0/1741026112 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1236836734 0 0) 0x7fcc64004eb0 con 0x7fcc8013c1b0 2024-03-29T11:28:48.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.721+0000 7fcc7d7fa640 1 -- 172.21.15.86:0/1741026112 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fcc8013e3c0 con 0x7fcc8013c1b0 2024-03-29T11:28:48.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.721+0000 7fcc7d7fa640 1 -- 172.21.15.86:0/1741026112 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fcc78001b50 con 0x7fcc8013c1b0 2024-03-29T11:28:48.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.721+0000 7fcc7d7fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:48.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.721+0000 7fcc7d7fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:48.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.721+0000 7fcc7d7fa640 10 monclient(hunting): dump: 2024-03-29T11:28:48.721 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:28:48.721 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:48.721 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:48.721 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:48.721 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:48.721 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:48.721 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:28:48.721 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:28:48.721 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:28:48.721 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:48.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.721+0000 7fcc7d7fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:48.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.722+0000 7fcc7d7fa640 1 -- 172.21.15.86:0/1741026112 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4151595127 0 0) 0x7fcc8013e3c0 con 0x7fcc8013c1b0 2024-03-29T11:28:48.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.722+0000 7fcc7d7fa640 1 -- 172.21.15.86:0/1741026112 >> v1:172.21.15.196:6789/0 conn(0x7fcc800b0770 legacy=0x7fcc801d2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:48.722 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.722+0000 7fcc7d7fa640 1 -- 172.21.15.86:0/1741026112 >> v1:172.21.15.86:6789/0 conn(0x7fcc800ad850 legacy=0x7fcc8013da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:48.722 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.722+0000 7fcc7d7fa640 10 monclient: _finish_hunting 0 2024-03-29T11:28:48.722 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.722+0000 7fcc7d7fa640 1 monclient: found mon.c 2024-03-29T11:28:48.722 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.722+0000 7fcc7d7fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:48.722 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.722+0000 7fcc7d7fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:28:48.722 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.722+0000 7fcc7d7fa640 1 -- 172.21.15.86:0/1741026112 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc8013fb50 con 0x7fcc8013c1b0 2024-03-29T11:28:48.722 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.722+0000 7fcc7d7fa640 10 monclient: _finish_auth 0 2024-03-29T11:28:48.722 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.722+0000 7fcc7d7fa640 10 monclient: _check_auth_tickets 2024-03-29T11:28:48.722 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.722+0000 7fcc7d7fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:48.722 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.722+0000 7fcc871d7640 5 monclient: authenticate success, global_id 4577 2024-03-29T11:28:48.722 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.722+0000 7fcc871d7640 10 monclient: _renew_subs 2024-03-29T11:28:48.722 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.722+0000 7fcc871d7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:28:48.722 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.722+0000 7fcc871d7640 1 -- 172.21.15.86:0/1741026112 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc8013fda0 con 0x7fcc8013c1b0 2024-03-29T11:28:48.722 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.722+0000 7fcc871d7640 10 client.4577.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:48.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.722+0000 7fcc871d7640 10 monclient: _renew_subs 2024-03-29T11:28:48.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.722+0000 7fcc871d7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:28:48.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.722+0000 7fcc871d7640 1 -- 172.21.15.86:0/1741026112 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc801d38d0 con 0x7fcc8013c1b0 2024-03-29T11:28:48.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.722+0000 7fcc871d7640 1 librados: init done 2024-03-29T11:28:48.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.723+0000 7fcc7d7fa640 1 -- 172.21.15.86:0/1741026112 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fcc78002080 con 0x7fcc8013c1b0 2024-03-29T11:28:48.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.723+0000 7fcc7d7fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:48.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.723+0000 7fcc7d7fa640 1 -- 172.21.15.86:0/1741026112 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fcc78004d00 con 0x7fcc8013c1b0 2024-03-29T11:28:48.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.723+0000 7fcc7d7fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:48.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.723+0000 7fcc7d7fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:48.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.723+0000 7fcc7d7fa640 10 monclient: dump: 2024-03-29T11:28:48.723 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:28:48.723 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:48.723 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:48.723 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:48.723 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:48.724 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:48.724 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:28:48.724 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:28:48.724 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:28:48.724 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:48.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.723+0000 7fcc7effd640 4 set_mon_vals no callback set 2024-03-29T11:28:48.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.724+0000 7fcc871d7640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:48.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.724+0000 7fcc871d7640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:48.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.724+0000 7fcc871d7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:28:48.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.724+0000 7fcc871d7640 1 -- 172.21.15.86:0/1741026112 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc60005080 con 0x7fcc8013c1b0 2024-03-29T11:28:48.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.725+0000 7fcc7d7fa640 1 -- 172.21.15.86:0/1741026112 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7fcc78004e60 con 0x7fcc8013c1b0 2024-03-29T11:28:48.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.725+0000 7fcc7d7fa640 1 -- 172.21.15.86:0/1741026112 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7fcc78084c90 con 0x7fcc8013c1b0 2024-03-29T11:28:48.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.725+0000 7fcc7d7fa640 10 client.4577.objecter ms_dispatch 0x7fcc80000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:48.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.725+0000 7fcc7d7fa640 3 client.4577.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:48.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.725+0000 7fcc7d7fa640 3 client.4577.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:48.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.725+0000 7fcc7d7fa640 20 client.4577.objecter dump_active .. 0 homeless 2024-03-29T11:28:48.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.726+0000 7fcc84f73640 10 monclient: get_auth_request con 0x7fcc6406dd80 auth_method 0 2024-03-29T11:28:48.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.727+0000 7fcc7d7fa640 10 client.4577.objecter ms_handle_connect 0x7fcc6406dd80 2024-03-29T11:28:48.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.732+0000 7fcc7d7fa640 1 -- 172.21.15.86:0/1741026112 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fcc78050570 con 0x7fcc8013c1b0 2024-03-29T11:28:48.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.732+0000 7fcc7d7fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:48.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:48.732+0000 7fcc7d7fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:49.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:49.054+0000 7fcc871d7640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:49.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:49.054+0000 7fcc871d7640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:49.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:49.054+0000 7fcc871d7640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:28:49.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:49.054+0000 7fcc871d7640 1 -- 172.21.15.86:0/1741026112 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcc60005330 con 0x7fcc8013c1b0 2024-03-29T11:28:49.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:49.055+0000 7fcc7d7fa640 1 -- 172.21.15.86:0/1741026112 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7fcc780524a0 con 0x7fcc8013c1b0 2024-03-29T11:28:49.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:49.055+0000 7fcc7d7fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:49.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:49.055+0000 7fcc7d7fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:49.048 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:28:49.048 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:49.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:49.058+0000 7fcc871d7640 10 librados: watch_flush enter 2024-03-29T11:28:49.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:49.060+0000 7fcc871d7640 10 librados: watch_flush exit 2024-03-29T11:28:49.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:49.060+0000 7fcc871d7640 20 client.4577.objecter shutdown clearing up homeless session... 2024-03-29T11:28:49.053 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:49.060+0000 7fcc871d7640 10 client.4577.objecter successfully canceled tick 2024-03-29T11:28:49.053 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:49.061+0000 7fcc871d7640 1 -- 172.21.15.86:0/1741026112 >> v1:172.21.15.196:6816/588110318 conn(0x7fcc6406dd80 legacy=0x7fcc64070240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:49.053 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:49.061+0000 7fcc871d7640 10 monclient: shutdown 2024-03-29T11:28:49.053 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:49.061+0000 7fcc871d7640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:49.053 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:49.061+0000 7fcc871d7640 1 -- 172.21.15.86:0/1741026112 >> v1:172.21.15.86:6790/0 conn(0x7fcc8013c1b0 legacy=0x7fcc801dd7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:49.053 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:49.061+0000 7fcc871d7640 1 -- 172.21.15.86:0/1741026112 shutdown_connections 2024-03-29T11:28:49.054 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:49.061+0000 7fcc871d7640 1 -- 172.21.15.86:0/1741026112 >> 172.21.15.86:0/1741026112 conn(0x7fcc8006b660 msgr2=0x7fcc800adee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:49.054 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:49.061+0000 7fcc871d7640 1 -- 172.21.15.86:0/1741026112 shutdown_connections 2024-03-29T11:28:49.054 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:49.061+0000 7fcc871d7640 1 -- 172.21.15.86:0/1741026112 wait complete. 2024-03-29T11:28:49.054 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:49.061+0000 7fcc871d7640 1 librados: shutdown 2024-03-29T11:28:49.068 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:28:55.876 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:28:56.022 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.031+0000 7f5d177ef640 10 monclient: get_monmap_and_config 2024-03-29T11:28:56.023 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.031+0000 7f5d177ef640 10 monclient: build_initial_monmap 2024-03-29T11:28:56.023 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.031+0000 7f5d177ef640 10 monclient: monmap: 2024-03-29T11:28:56.023 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:28:56.023 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:56.023 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:28:56.032887+0000 2024-03-29T11:28:56.023 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:28:56.032887+0000 2024-03-29T11:28:56.023 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:56.023 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:56.023 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:28:56.023 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:28:56.023 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:28:56.023 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:56.024 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.033+0000 7f5d177ef640 1 Processor -- start 2024-03-29T11:28:56.024 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.033+0000 7f5d177ef640 1 -- start start 2024-03-29T11:28:56.024 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.033+0000 7f5d177ef640 10 monclient: init 2024-03-29T11:28:56.026 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.034+0000 7f5d177ef640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:56.026 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.034+0000 7f5d177ef640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:28:56.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.034+0000 7f5d177ef640 10 monclient(hunting): picked mon.noname-b con 0x7f5d100b0770 addr v1:172.21.15.86:6790/0 2024-03-29T11:28:56.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.034+0000 7f5d177ef640 10 monclient(hunting): picked mon.noname-c con 0x7f5d1014a700 addr v1:172.21.15.196:6789/0 2024-03-29T11:28:56.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.035+0000 7f5d177ef640 10 monclient(hunting): picked mon.noname-a con 0x7f5d100ac5a0 addr v1:172.21.15.86:6789/0 2024-03-29T11:28:56.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.035+0000 7f5d177ef640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d100aba70 con 0x7f5d100ac5a0 2024-03-29T11:28:56.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.035+0000 7f5d177ef640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d100aecb0 con 0x7f5d1014a700 2024-03-29T11:28:56.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.035+0000 7f5d177ef640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d100af150 con 0x7f5d100b0770 2024-03-29T11:28:56.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.035+0000 7f5d177ef640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:56.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.035+0000 7f5d177ef640 10 monclient(hunting): authenticate will time out at 1360.353070s 2024-03-29T11:28:56.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.035+0000 7f5d1558b640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f5d100b0770 0x7f5d10149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:42498/0 (socket says 172.21.15.86:42498) 2024-03-29T11:28:56.027 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.035+0000 7f5d1558b640 1 -- 172.21.15.86:0/1834621419 learned_addr learned my addr 172.21.15.86:0/1834621419 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:28:56.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.037+0000 7f5d07fff640 1 -- 172.21.15.86:0/1834621419 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1214387569 0 0) 0x7f5d100af150 con 0x7f5d100b0770 2024-03-29T11:28:56.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.037+0000 7f5d07fff640 10 monclient(hunting): _negotiate 2024-03-29T11:28:56.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.037+0000 7f5d07fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:56.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.037+0000 7f5d07fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:56.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.037+0000 7f5d07fff640 10 monclient(hunting): my global_id is 4580 2024-03-29T11:28:56.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.037+0000 7f5d07fff640 1 -- 172.21.15.86:0/1834621419 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cfc003520 con 0x7f5d100b0770 2024-03-29T11:28:56.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.038+0000 7f5d07fff640 1 -- 172.21.15.86:0/1834621419 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2576188432 0 0) 0x7f5cfc003520 con 0x7f5d100b0770 2024-03-29T11:28:56.029 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.038+0000 7f5d07fff640 1 -- 172.21.15.86:0/1834621419 >> v1:172.21.15.196:6789/0 conn(0x7f5d1014a700 legacy=0x7f5d100abe90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:28:56.032 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.040+0000 7f5d07fff640 1 -- 172.21.15.86:0/1834621419 >> v1:172.21.15.86:6789/0 conn(0x7f5d100ac5a0 legacy=0x7f5d10151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:56.032 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.040+0000 7f5d07fff640 10 monclient: _finish_hunting 0 2024-03-29T11:28:56.032 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.040+0000 7f5d07fff640 1 monclient: found mon.noname-b 2024-03-29T11:28:56.032 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.040+0000 7f5d07fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:56.032 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.040+0000 7f5d07fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:28:56.032 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.040+0000 7f5d07fff640 1 -- 172.21.15.86:0/1834621419 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d100ac9d0 con 0x7f5d100b0770 2024-03-29T11:28:56.033 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.040+0000 7f5d07fff640 10 monclient: _finish_auth 0 2024-03-29T11:28:56.033 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.040+0000 7f5d07fff640 10 monclient: _check_auth_tickets 2024-03-29T11:28:56.033 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.040+0000 7f5d07fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:56.033 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.040+0000 7f5d07fff640 1 -- 172.21.15.86:0/1834621419 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f5d08001a10 con 0x7f5d100b0770 2024-03-29T11:28:56.033 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.040+0000 7f5d07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:56.033 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.040+0000 7f5d07fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:28:56.033 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.040+0000 7f5d07fff640 10 monclient: dump: 2024-03-29T11:28:56.033 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:28:56.033 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:56.033 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:56.033 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:56.034 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:56.034 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:56.034 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:28:56.034 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:28:56.034 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:28:56.034 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:56.034 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.040+0000 7f5d177ef640 5 monclient: authenticate success, global_id 4580 2024-03-29T11:28:56.034 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.040+0000 7f5d177ef640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:28:56.034 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.040+0000 7f5d07fff640 1 -- 172.21.15.86:0/1834621419 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d08001be0 con 0x7f5d100b0770 2024-03-29T11:28:56.034 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.040+0000 7f5d07fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:56.034 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.040+0000 7f5d07fff640 1 -- 172.21.15.86:0/1834621419 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f5d08002030 con 0x7f5d100b0770 2024-03-29T11:28:56.034 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.040+0000 7f5d177ef640 10 monclient: get_monmap_and_config success 2024-03-29T11:28:56.034 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.040+0000 7f5d177ef640 4 set_mon_vals no callback set 2024-03-29T11:28:56.034 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.040+0000 7f5d07fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:56.034 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.040+0000 7f5d07fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:56.035 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.040+0000 7f5d07fff640 10 monclient: dump: 2024-03-29T11:28:56.035 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:28:56.035 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:56.035 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:56.035 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:56.035 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:56.035 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:56.035 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:28:56.035 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:28:56.035 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:28:56.035 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:56.035 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.041+0000 7f5d177ef640 10 monclient: shutdown 2024-03-29T11:28:56.035 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.041+0000 7f5d177ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:56.035 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.041+0000 7f5d177ef640 1 -- 172.21.15.86:0/1834621419 >> v1:172.21.15.86:6790/0 conn(0x7f5d100b0770 legacy=0x7f5d10149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:56.036 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.041+0000 7f5d177ef640 1 -- 172.21.15.86:0/1834621419 shutdown_connections 2024-03-29T11:28:56.036 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.041+0000 7f5d177ef640 1 -- 172.21.15.86:0/1834621419 >> 172.21.15.86:0/1834621419 conn(0x7f5d1006b660 msgr2=0x7f5d1006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:56.036 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.042+0000 7f5d177ef640 1 -- 172.21.15.86:0/1834621419 shutdown_connections 2024-03-29T11:28:56.036 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.042+0000 7f5d177ef640 1 -- 172.21.15.86:0/1834621419 wait complete. 2024-03-29T11:28:56.043 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.043+0000 7f5d177ef640 10 monclient: build_initial_monmap 2024-03-29T11:28:56.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.043+0000 7f5d177ef640 10 monclient: monmap: 2024-03-29T11:28:56.044 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:28:56.044 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:28:56.044 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:28:56.044 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:28:56.044 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:28:56.044 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:56.044 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:28:56.044 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:28:56.044 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:28:56.044 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:56.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.043+0000 7f5d177ef640 1 librados: starting msgr at 2024-03-29T11:28:56.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.043+0000 7f5d177ef640 1 librados: starting objecter 2024-03-29T11:28:56.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.043+0000 7f5d177ef640 1 Processor -- start 2024-03-29T11:28:56.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.044+0000 7f5d177ef640 1 -- start start 2024-03-29T11:28:56.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.044+0000 7f5d177ef640 1 librados: setting wanted keys 2024-03-29T11:28:56.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.044+0000 7f5d177ef640 1 librados: calling monclient init 2024-03-29T11:28:56.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.044+0000 7f5d177ef640 10 monclient: init 2024-03-29T11:28:56.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.045+0000 7f5d177ef640 10 monclient: _reopen_session rank -1 2024-03-29T11:28:56.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.045+0000 7f5d177ef640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:28:56.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.045+0000 7f5d177ef640 10 monclient(hunting): picked mon.noname-a con 0x7f5d100ac5a0 addr v1:172.21.15.86:6789/0 2024-03-29T11:28:56.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.045+0000 7f5d177ef640 10 monclient(hunting): picked mon.noname-b con 0x7f5d100b0770 addr v1:172.21.15.196:6789/0 2024-03-29T11:28:56.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.045+0000 7f5d177ef640 10 monclient(hunting): picked mon.noname-c con 0x7f5d1014a700 addr v1:172.21.15.86:6790/0 2024-03-29T11:28:56.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.045+0000 7f5d177ef640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d101201a0 con 0x7f5d100ac5a0 2024-03-29T11:28:56.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.045+0000 7f5d177ef640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d101229b0 con 0x7f5d100b0770 2024-03-29T11:28:56.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.045+0000 7f5d177ef640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d10121ba0 con 0x7f5d1014a700 2024-03-29T11:28:56.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.045+0000 7f5d177ef640 10 monclient(hunting): _renew_subs 2024-03-29T11:28:56.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.045+0000 7f5d177ef640 10 monclient(hunting): authenticate will time out at 1360.363583s 2024-03-29T11:28:56.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.045+0000 7f5d15d8c640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f5d1014a700 0x7f5d10146fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:42926/0 (socket says 172.21.15.86:42926) 2024-03-29T11:28:56.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.045+0000 7f5d15d8c640 1 -- 172.21.15.86:0/1928855674 learned_addr learned my addr 172.21.15.86:0/1928855674 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:28:56.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.046+0000 7f5d05ffb640 10 client.?.objecter ms_handle_connect 0x7f5d1014a700 2024-03-29T11:28:56.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.046+0000 7f5d05ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:28:56.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.046+0000 7f5d05ffb640 1 -- 172.21.15.86:0/1928855674 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924145681 0 0) 0x7f5d10121ba0 con 0x7f5d1014a700 2024-03-29T11:28:56.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.046+0000 7f5d05ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:28:56.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.046+0000 7f5d05ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:28:56.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.046+0000 7f5d05ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:28:56.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.046+0000 7f5d05ffb640 10 monclient(hunting): my global_id is 4583 2024-03-29T11:28:56.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.046+0000 7f5d05ffb640 1 -- 172.21.15.86:0/1928855674 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cf4004eb0 con 0x7f5d1014a700 2024-03-29T11:28:56.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.046+0000 7f5d05ffb640 1 -- 172.21.15.86:0/1928855674 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2917137215 0 0) 0x7f5cf4004eb0 con 0x7f5d1014a700 2024-03-29T11:28:56.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.046+0000 7f5d05ffb640 1 -- 172.21.15.86:0/1928855674 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f5d10121ba0 con 0x7f5d1014a700 2024-03-29T11:28:56.046 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.047+0000 7f5d05ffb640 1 -- 172.21.15.86:0/1928855674 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f5d0c0018b0 con 0x7f5d1014a700 2024-03-29T11:28:56.047 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.047+0000 7f5d05ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:56.047 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.047+0000 7f5d05ffb640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:28:56.047 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.047+0000 7f5d05ffb640 10 monclient(hunting): dump: 2024-03-29T11:28:56.047 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:28:56.047 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:56.047 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:56.047 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:56.047 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:56.047 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:56.047 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:28:56.047 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:28:56.047 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:28:56.047 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:56.047 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.047+0000 7f5d05ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:28:56.047 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.047+0000 7f5d05ffb640 1 -- 172.21.15.86:0/1928855674 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2232487055 0 0) 0x7f5d10121ba0 con 0x7f5d1014a700 2024-03-29T11:28:56.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.047+0000 7f5d05ffb640 1 -- 172.21.15.86:0/1928855674 >> v1:172.21.15.196:6789/0 conn(0x7f5d100b0770 legacy=0x7f5d101222a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:28:56.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.048+0000 7f5d05ffb640 1 -- 172.21.15.86:0/1928855674 >> v1:172.21.15.86:6789/0 conn(0x7f5d100ac5a0 legacy=0x7f5d1011fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:56.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.049+0000 7f5d05ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:28:56.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.049+0000 7f5d05ffb640 1 monclient: found mon.c 2024-03-29T11:28:56.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.049+0000 7f5d05ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:28:56.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.049+0000 7f5d05ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:28:56.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.049+0000 7f5d05ffb640 1 -- 172.21.15.86:0/1928855674 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d10121e80 con 0x7f5d1014a700 2024-03-29T11:28:56.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.049+0000 7f5d05ffb640 10 monclient: _finish_auth 0 2024-03-29T11:28:56.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.049+0000 7f5d05ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:28:56.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.049+0000 7f5d05ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:28:56.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.049+0000 7f5d05ffb640 10 client.?.objecter ms_handle_connect 0x7f5d100ac5a0 2024-03-29T11:28:56.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.049+0000 7f5d177ef640 5 monclient: authenticate success, global_id 4583 2024-03-29T11:28:56.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.049+0000 7f5d177ef640 10 monclient: _renew_subs 2024-03-29T11:28:56.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.049+0000 7f5d177ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:28:56.048 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.049+0000 7f5d177ef640 1 -- 172.21.15.86:0/1928855674 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d10147c30 con 0x7f5d1014a700 2024-03-29T11:28:56.049 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.050+0000 7f5d177ef640 10 client.4583.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:28:56.049 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.050+0000 7f5d177ef640 10 monclient: _renew_subs 2024-03-29T11:28:56.049 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.050+0000 7f5d177ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:28:56.049 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.050+0000 7f5d177ef640 1 -- 172.21.15.86:0/1928855674 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d101478d0 con 0x7f5d1014a700 2024-03-29T11:28:56.049 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.050+0000 7f5d177ef640 1 librados: init done 2024-03-29T11:28:56.049 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.050+0000 7f5d05ffb640 10 client.4583.objecter resend_mon_ops 2024-03-29T11:28:56.049 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.050+0000 7f5d05ffb640 1 -- 172.21.15.86:0/1928855674 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f5d0c001a80 con 0x7f5d1014a700 2024-03-29T11:28:56.049 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.050+0000 7f5d05ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:28:56.049 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.050+0000 7f5d05ffb640 1 -- 172.21.15.86:0/1928855674 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f5d0c0038c0 con 0x7f5d1014a700 2024-03-29T11:28:56.049 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.050+0000 7f5d05ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:28:56.049 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.050+0000 7f5d05ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:28:56.049 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.050+0000 7f5d05ffb640 10 monclient: dump: 2024-03-29T11:28:56.049 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:28:56.049 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:28:56.049 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:56.050 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:28:56.050 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:28:56.050 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:28:56.050 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:28:56.050 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:28:56.050 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:28:56.050 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:28:56.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.050+0000 7f5d06ffd640 4 set_mon_vals no callback set 2024-03-29T11:28:56.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.052+0000 7f5d177ef640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:56.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.052+0000 7f5d05ffb640 1 -- 172.21.15.86:0/1928855674 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f5d0c003a20 con 0x7f5d1014a700 2024-03-29T11:28:56.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.052+0000 7f5d177ef640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:56.051 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.052+0000 7f5d177ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:28:56.051 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.052+0000 7f5d177ef640 1 -- 172.21.15.86:0/1928855674 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d100b0b50 con 0x7f5d1014a700 2024-03-29T11:28:56.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.053+0000 7f5d1558b640 10 monclient: get_auth_request con 0x7f5cf406dc10 auth_method 0 2024-03-29T11:28:56.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.054+0000 7f5d05ffb640 1 -- 172.21.15.86:0/1928855674 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f5d0c084a80 con 0x7f5d1014a700 2024-03-29T11:28:56.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.054+0000 7f5d05ffb640 10 client.4583.objecter ms_dispatch 0x7f5d10000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:28:56.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.054+0000 7f5d05ffb640 3 client.4583.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:28:56.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.054+0000 7f5d05ffb640 3 client.4583.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:28:56.053 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.054+0000 7f5d05ffb640 20 client.4583.objecter dump_active .. 0 homeless 2024-03-29T11:28:56.053 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.054+0000 7f5d05ffb640 10 client.4583.objecter ms_handle_connect 0x7f5cf406dc10 2024-03-29T11:28:56.053 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.061+0000 7f5d05ffb640 1 -- 172.21.15.86:0/1928855674 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f5d0c050360 con 0x7f5d1014a700 2024-03-29T11:28:56.053 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.061+0000 7f5d05ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:28:56.053 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.061+0000 7f5d05ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:28:56.376 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.385+0000 7f5d177ef640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:56.376 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.386+0000 7f5d177ef640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:56.377 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.386+0000 7f5d177ef640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:28:56.377 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.386+0000 7f5d177ef640 1 -- 172.21.15.86:0/1928855674 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d10075f10 con 0x7f5d1014a700 2024-03-29T11:28:56.377 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.386+0000 7f5d05ffb640 1 -- 172.21.15.86:0/1928855674 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f5d0c052290 con 0x7f5d1014a700 2024-03-29T11:28:56.377 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.386+0000 7f5d05ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:28:56.377 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.386+0000 7f5d05ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:28:56.378 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:28:56.378 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:28:56.381 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.390+0000 7f5d177ef640 10 librados: watch_flush enter 2024-03-29T11:28:56.381 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.390+0000 7f5d177ef640 10 librados: watch_flush exit 2024-03-29T11:28:56.381 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.390+0000 7f5d177ef640 20 client.4583.objecter shutdown clearing up homeless session... 2024-03-29T11:28:56.381 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.390+0000 7f5d177ef640 10 client.4583.objecter successfully canceled tick 2024-03-29T11:28:56.381 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.390+0000 7f5d177ef640 1 -- 172.21.15.86:0/1928855674 >> v1:172.21.15.196:6816/588110318 conn(0x7f5cf406dc10 legacy=0x7f5cf40700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:56.381 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.390+0000 7f5d177ef640 10 monclient: shutdown 2024-03-29T11:28:56.381 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.390+0000 7f5d177ef640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:28:56.381 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.390+0000 7f5d177ef640 1 -- 172.21.15.86:0/1928855674 >> v1:172.21.15.86:6790/0 conn(0x7f5d1014a700 legacy=0x7f5d10146fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:28:56.381 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.390+0000 7f5d177ef640 1 -- 172.21.15.86:0/1928855674 shutdown_connections 2024-03-29T11:28:56.382 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.391+0000 7f5d177ef640 1 -- 172.21.15.86:0/1928855674 >> 172.21.15.86:0/1928855674 conn(0x7f5d1006b660 msgr2=0x7f5d100b2e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:28:56.382 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.391+0000 7f5d177ef640 1 -- 172.21.15.86:0/1928855674 shutdown_connections 2024-03-29T11:28:56.382 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.391+0000 7f5d177ef640 1 -- 172.21.15.86:0/1928855674 wait complete. 2024-03-29T11:28:56.382 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:28:56.391+0000 7f5d177ef640 1 librados: shutdown 2024-03-29T11:28:56.398 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:03.205 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:03.351 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.359+0000 7f357470f640 10 monclient: get_monmap_and_config 2024-03-29T11:29:03.351 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.359+0000 7f357470f640 10 monclient: build_initial_monmap 2024-03-29T11:29:03.351 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.360+0000 7f357470f640 10 monclient: monmap: 2024-03-29T11:29:03.351 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:29:03.352 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:03.352 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:29:03.360977+0000 2024-03-29T11:29:03.352 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:29:03.360977+0000 2024-03-29T11:29:03.352 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:03.352 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:03.352 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:29:03.352 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:29:03.352 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:29:03.352 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:03.354 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.361+0000 7f357470f640 1 Processor -- start 2024-03-29T11:29:03.354 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.361+0000 7f357470f640 1 -- start start 2024-03-29T11:29:03.354 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.361+0000 7f357470f640 10 monclient: init 2024-03-29T11:29:03.354 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.362+0000 7f357470f640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:03.354 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.362+0000 7f357470f640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:29:03.354 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.362+0000 7f357470f640 10 monclient(hunting): picked mon.noname-b con 0x7f356c0b4100 addr v1:172.21.15.86:6790/0 2024-03-29T11:29:03.354 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.362+0000 7f357470f640 10 monclient(hunting): picked mon.noname-a con 0x7f356c13ee90 addr v1:172.21.15.86:6789/0 2024-03-29T11:29:03.354 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.362+0000 7f357470f640 10 monclient(hunting): picked mon.noname-c con 0x7f356c0a9ac0 addr v1:172.21.15.196:6789/0 2024-03-29T11:29:03.354 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.362+0000 7f357470f640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f356c0af6f0 con 0x7f356c13ee90 2024-03-29T11:29:03.354 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.362+0000 7f357470f640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f356c0aac80 con 0x7f356c0a9ac0 2024-03-29T11:29:03.355 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.362+0000 7f357470f640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f356c0aaf60 con 0x7f356c0b4100 2024-03-29T11:29:03.355 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.362+0000 7f357470f640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:03.355 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.362+0000 7f357470f640 10 monclient(hunting): authenticate will time out at 1367.680325s 2024-03-29T11:29:03.355 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.362+0000 7f35724ab640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f356c0b4100 0x7f356c0adfe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:42946/0 (socket says 172.21.15.86:42946) 2024-03-29T11:29:03.355 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.362+0000 7f35724ab640 1 -- 172.21.15.86:0/207044764 learned_addr learned my addr 172.21.15.86:0/207044764 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:29:03.363 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.366+0000 7f35714a9640 1 -- 172.21.15.86:0/207044764 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4070819981 0 0) 0x7f356c0aaf60 con 0x7f356c0b4100 2024-03-29T11:29:03.363 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.367+0000 7f35714a9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:03.363 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.367+0000 7f35714a9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:03.363 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.367+0000 7f35714a9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:03.363 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.367+0000 7f35714a9640 10 monclient(hunting): my global_id is 4586 2024-03-29T11:29:03.363 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.367+0000 7f35714a9640 1 -- 172.21.15.86:0/207044764 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3558003520 con 0x7f356c0b4100 2024-03-29T11:29:03.363 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.367+0000 7f35714a9640 1 -- 172.21.15.86:0/207044764 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 745874363 0 0) 0x7f3558003520 con 0x7f356c0b4100 2024-03-29T11:29:03.363 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.367+0000 7f35714a9640 1 -- 172.21.15.86:0/207044764 >> v1:172.21.15.196:6789/0 conn(0x7f356c0a9ac0 legacy=0x7f356c142ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:03.363 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.367+0000 7f35714a9640 1 -- 172.21.15.86:0/207044764 >> v1:172.21.15.86:6789/0 conn(0x7f356c13ee90 legacy=0x7f356c13b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:03.364 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.367+0000 7f35714a9640 10 monclient: _finish_hunting 0 2024-03-29T11:29:03.364 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.367+0000 7f35714a9640 1 monclient: found mon.noname-b 2024-03-29T11:29:03.364 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.367+0000 7f35714a9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:03.364 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.367+0000 7f35714a9640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:29:03.364 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.367+0000 7f35714a9640 1 -- 172.21.15.86:0/207044764 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f356c13e970 con 0x7f356c0b4100 2024-03-29T11:29:03.364 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.367+0000 7f35714a9640 10 monclient: _finish_auth 0 2024-03-29T11:29:03.364 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.367+0000 7f35714a9640 10 monclient: _check_auth_tickets 2024-03-29T11:29:03.364 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.367+0000 7f35714a9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:03.364 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.367+0000 7f35714a9640 1 -- 172.21.15.86:0/207044764 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f3568001690 con 0x7f356c0b4100 2024-03-29T11:29:03.364 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.367+0000 7f35714a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:03.364 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.367+0000 7f35714a9640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:03.364 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.367+0000 7f35714a9640 10 monclient: dump: 2024-03-29T11:29:03.364 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:29:03.364 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:03.365 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:03.365 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:03.365 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:03.365 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:03.365 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:29:03.365 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:29:03.365 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:29:03.365 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:03.365 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.367+0000 7f35714a9640 1 -- 172.21.15.86:0/207044764 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3568001860 con 0x7f356c0b4100 2024-03-29T11:29:03.365 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.367+0000 7f35714a9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:03.365 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.367+0000 7f35714a9640 1 -- 172.21.15.86:0/207044764 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f3568001cb0 con 0x7f356c0b4100 2024-03-29T11:29:03.365 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.367+0000 7f35714a9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:03.365 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.367+0000 7f35714a9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:03.366 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.367+0000 7f35714a9640 10 monclient: dump: 2024-03-29T11:29:03.366 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:29:03.366 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:03.366 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:03.366 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:03.366 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:03.366 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:03.366 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:29:03.366 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:29:03.366 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:29:03.366 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:03.367 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.368+0000 7f357470f640 5 monclient: authenticate success, global_id 4586 2024-03-29T11:29:03.367 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.368+0000 7f357470f640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:03.367 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.368+0000 7f357470f640 4 set_mon_vals no callback set 2024-03-29T11:29:03.367 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.370+0000 7f357470f640 10 monclient: shutdown 2024-03-29T11:29:03.367 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.370+0000 7f357470f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:03.367 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.370+0000 7f357470f640 1 -- 172.21.15.86:0/207044764 >> v1:172.21.15.86:6790/0 conn(0x7f356c0b4100 legacy=0x7f356c0adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:03.367 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.371+0000 7f357470f640 1 -- 172.21.15.86:0/207044764 shutdown_connections 2024-03-29T11:29:03.367 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.371+0000 7f357470f640 1 -- 172.21.15.86:0/207044764 >> 172.21.15.86:0/207044764 conn(0x7f356c0b35e0 msgr2=0x7f356c0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:03.374 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.373+0000 7f357470f640 1 -- 172.21.15.86:0/207044764 shutdown_connections 2024-03-29T11:29:03.374 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.373+0000 7f357470f640 1 -- 172.21.15.86:0/207044764 wait complete. 2024-03-29T11:29:03.374 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.374+0000 7f357470f640 10 monclient: build_initial_monmap 2024-03-29T11:29:03.374 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.374+0000 7f357470f640 10 monclient: monmap: 2024-03-29T11:29:03.374 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:29:03.374 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:03.374 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:29:03.374 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:29:03.374 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:03.375 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:03.375 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:29:03.375 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:29:03.375 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:29:03.375 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:03.375 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.374+0000 7f357470f640 1 librados: starting msgr at 2024-03-29T11:29:03.375 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.374+0000 7f357470f640 1 librados: starting objecter 2024-03-29T11:29:03.375 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.374+0000 7f357470f640 1 Processor -- start 2024-03-29T11:29:03.375 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.374+0000 7f357470f640 1 -- start start 2024-03-29T11:29:03.375 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.374+0000 7f357470f640 1 librados: setting wanted keys 2024-03-29T11:29:03.375 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.374+0000 7f357470f640 1 librados: calling monclient init 2024-03-29T11:29:03.375 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.374+0000 7f357470f640 10 monclient: init 2024-03-29T11:29:03.375 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.375+0000 7f357470f640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:03.376 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.375+0000 7f357470f640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:03.376 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.375+0000 7f357470f640 10 monclient(hunting): picked mon.noname-b con 0x7f356c0a9ac0 addr v1:172.21.15.196:6789/0 2024-03-29T11:29:03.376 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.375+0000 7f357470f640 10 monclient(hunting): picked mon.noname-c con 0x7f356c0b4100 addr v1:172.21.15.86:6790/0 2024-03-29T11:29:03.376 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.375+0000 7f357470f640 10 monclient(hunting): picked mon.noname-a con 0x7f356c13ee90 addr v1:172.21.15.86:6789/0 2024-03-29T11:29:03.376 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.375+0000 7f357470f640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f356c0701f0 con 0x7f356c13ee90 2024-03-29T11:29:03.376 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.375+0000 7f357470f640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f356c0704d0 con 0x7f356c0a9ac0 2024-03-29T11:29:03.376 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.375+0000 7f357470f640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f356c06ef90 con 0x7f356c0b4100 2024-03-29T11:29:03.376 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.375+0000 7f357470f640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:03.376 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.375+0000 7f357470f640 10 monclient(hunting): authenticate will time out at 1367.693577s 2024-03-29T11:29:03.376 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.378+0000 7f3571caa640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f356c0b4100 0x7f356c139990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:43454/0 (socket says 172.21.15.86:43454) 2024-03-29T11:29:03.376 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.378+0000 7f3571caa640 1 -- 172.21.15.86:0/69879347 learned_addr learned my addr 172.21.15.86:0/69879347 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:29:03.377 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.379+0000 7f3562ffd640 10 client.?.objecter ms_handle_connect 0x7f356c0b4100 2024-03-29T11:29:03.377 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.379+0000 7f3562ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:03.377 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.379+0000 7f3562ffd640 1 -- 172.21.15.86:0/69879347 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1421652356 0 0) 0x7f356c06ef90 con 0x7f356c0b4100 2024-03-29T11:29:03.377 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.379+0000 7f3562ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:03.377 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.379+0000 7f3562ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:03.377 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.379+0000 7f3562ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:03.377 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.379+0000 7f3562ffd640 10 monclient(hunting): my global_id is 4589 2024-03-29T11:29:03.377 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.379+0000 7f3562ffd640 1 -- 172.21.15.86:0/69879347 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3550004eb0 con 0x7f356c0b4100 2024-03-29T11:29:03.377 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.379+0000 7f3562ffd640 1 -- 172.21.15.86:0/69879347 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 328399315 0 0) 0x7f3550004eb0 con 0x7f356c0b4100 2024-03-29T11:29:03.377 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.379+0000 7f3562ffd640 1 -- 172.21.15.86:0/69879347 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f356c06ef90 con 0x7f356c0b4100 2024-03-29T11:29:03.377 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.379+0000 7f3562ffd640 1 -- 172.21.15.86:0/69879347 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f355c0017d0 con 0x7f356c0b4100 2024-03-29T11:29:03.377 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.379+0000 7f3562ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:03.377 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.379+0000 7f3562ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:03.377 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.379+0000 7f3562ffd640 10 monclient(hunting): dump: 2024-03-29T11:29:03.378 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:29:03.378 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:03.378 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:03.378 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:03.378 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:03.378 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:03.378 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:29:03.378 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:29:03.378 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:29:03.378 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:03.378 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.379+0000 7f3562ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:03.378 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.379+0000 7f3562ffd640 10 client.?.objecter ms_handle_connect 0x7f356c13ee90 2024-03-29T11:29:03.378 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.379+0000 7f3562ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:03.378 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.379+0000 7f3562ffd640 1 -- 172.21.15.86:0/69879347 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2435248888 0 0) 0x7f356c0701f0 con 0x7f356c13ee90 2024-03-29T11:29:03.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.379+0000 7f3562ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:03.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.379+0000 7f3562ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:03.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.379+0000 7f3562ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:03.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.379+0000 7f3562ffd640 10 monclient(hunting): my global_id is 4506 2024-03-29T11:29:03.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.379+0000 7f3562ffd640 1 -- 172.21.15.86:0/69879347 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3550004eb0 con 0x7f356c13ee90 2024-03-29T11:29:03.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.380+0000 7f3562ffd640 1 -- 172.21.15.86:0/69879347 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3725219076 0 0) 0x7f356c06ef90 con 0x7f356c0b4100 2024-03-29T11:29:03.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.380+0000 7f3562ffd640 1 -- 172.21.15.86:0/69879347 >> v1:172.21.15.196:6789/0 conn(0x7f356c0a9ac0 legacy=0x7f356c06e7a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:29:03.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.381+0000 7f3562ffd640 1 -- 172.21.15.86:0/69879347 >> v1:172.21.15.86:6789/0 conn(0x7f356c13ee90 legacy=0x7f356c070960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:03.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.381+0000 7f3562ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:03.379 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.381+0000 7f3562ffd640 1 monclient: found mon.c 2024-03-29T11:29:03.380 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.381+0000 7f3562ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:03.380 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.381+0000 7f3562ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:29:03.380 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.381+0000 7f3562ffd640 1 -- 172.21.15.86:0/69879347 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f356c13a0a0 con 0x7f356c0b4100 2024-03-29T11:29:03.380 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.381+0000 7f3562ffd640 10 monclient: _finish_auth 0 2024-03-29T11:29:03.380 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.381+0000 7f3562ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:03.380 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.381+0000 7f3562ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:03.380 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.382+0000 7f3562ffd640 1 -- 172.21.15.86:0/69879347 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f355c001930 con 0x7f356c0b4100 2024-03-29T11:29:03.380 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.382+0000 7f3562ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:03.380 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.382+0000 7f3570ca8640 4 set_mon_vals no callback set 2024-03-29T11:29:03.380 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.382+0000 7f3562ffd640 1 -- 172.21.15.86:0/69879347 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f355c004a40 con 0x7f356c0b4100 2024-03-29T11:29:03.380 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.382+0000 7f3562ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:03.380 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.382+0000 7f3562ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:03.380 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.382+0000 7f3562ffd640 10 monclient: dump: 2024-03-29T11:29:03.381 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:29:03.381 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:03.381 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:03.381 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:03.381 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:03.381 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:03.381 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:29:03.381 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:29:03.381 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:29:03.381 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:03.381 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.383+0000 7f357470f640 5 monclient: authenticate success, global_id 4589 2024-03-29T11:29:03.381 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.383+0000 7f357470f640 10 monclient: _renew_subs 2024-03-29T11:29:03.381 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.383+0000 7f357470f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:29:03.381 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.383+0000 7f357470f640 1 -- 172.21.15.86:0/69879347 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f356c1421a0 con 0x7f356c0b4100 2024-03-29T11:29:03.383 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.384+0000 7f357470f640 10 client.4589.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:03.384 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.384+0000 7f357470f640 10 monclient: _renew_subs 2024-03-29T11:29:03.384 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.384+0000 7f357470f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:29:03.384 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.384+0000 7f357470f640 1 -- 172.21.15.86:0/69879347 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f356c071070 con 0x7f356c0b4100 2024-03-29T11:29:03.384 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.384+0000 7f357470f640 1 librados: init done 2024-03-29T11:29:03.384 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.387+0000 7f3562ffd640 1 -- 172.21.15.86:0/69879347 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f355c01aa10 con 0x7f356c0b4100 2024-03-29T11:29:03.384 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.387+0000 7f357470f640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:03.384 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.387+0000 7f357470f640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:03.384 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.387+0000 7f357470f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:29:03.384 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.387+0000 7f357470f640 1 -- 172.21.15.86:0/69879347 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f356c13a290 con 0x7f356c0b4100 2024-03-29T11:29:03.384 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.388+0000 7f3562ffd640 1 -- 172.21.15.86:0/69879347 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f355c0548d0 con 0x7f356c0b4100 2024-03-29T11:29:03.385 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.388+0000 7f3562ffd640 10 client.4589.objecter ms_dispatch 0x7f356c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:03.385 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.388+0000 7f3562ffd640 3 client.4589.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:03.385 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.388+0000 7f3562ffd640 3 client.4589.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:03.385 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.388+0000 7f3562ffd640 20 client.4589.objecter dump_active .. 0 homeless 2024-03-29T11:29:03.385 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.388+0000 7f35724ab640 10 monclient: get_auth_request con 0x7f355006daa0 auth_method 0 2024-03-29T11:29:03.385 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.388+0000 7f3562ffd640 10 client.4589.objecter ms_handle_connect 0x7f355006daa0 2024-03-29T11:29:03.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.395+0000 7f3562ffd640 1 -- 172.21.15.86:0/69879347 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f355c050210 con 0x7f356c0b4100 2024-03-29T11:29:03.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.395+0000 7f3562ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:03.386 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.395+0000 7f3562ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:03.709 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.719+0000 7f357470f640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:03.710 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.719+0000 7f357470f640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:03.710 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.719+0000 7f357470f640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:29:03.710 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.719+0000 7f357470f640 1 -- 172.21.15.86:0/69879347 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f356c0af900 con 0x7f356c0b4100 2024-03-29T11:29:03.710 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.719+0000 7f3562ffd640 1 -- 172.21.15.86:0/69879347 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f356c0af900 con 0x7f356c0b4100 2024-03-29T11:29:03.710 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.719+0000 7f3562ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:03.710 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.719+0000 7f3562ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:03.711 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:29:03.711 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:03.714 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.723+0000 7f357470f640 10 librados: watch_flush enter 2024-03-29T11:29:03.714 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.723+0000 7f357470f640 10 librados: watch_flush exit 2024-03-29T11:29:03.714 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.723+0000 7f357470f640 20 client.4589.objecter shutdown clearing up homeless session... 2024-03-29T11:29:03.714 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.723+0000 7f357470f640 10 client.4589.objecter successfully canceled tick 2024-03-29T11:29:03.714 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.723+0000 7f357470f640 1 -- 172.21.15.86:0/69879347 >> v1:172.21.15.196:6816/588110318 conn(0x7f355006daa0 legacy=0x7f355006ff60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:03.714 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.723+0000 7f357470f640 10 monclient: shutdown 2024-03-29T11:29:03.714 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.723+0000 7f357470f640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:03.714 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.723+0000 7f357470f640 1 -- 172.21.15.86:0/69879347 >> v1:172.21.15.86:6790/0 conn(0x7f356c0b4100 legacy=0x7f356c139990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:03.714 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.723+0000 7f357470f640 1 -- 172.21.15.86:0/69879347 shutdown_connections 2024-03-29T11:29:03.715 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.723+0000 7f357470f640 1 -- 172.21.15.86:0/69879347 >> 172.21.15.86:0/69879347 conn(0x7f356c0b35e0 msgr2=0x7f356c13e1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:03.715 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.723+0000 7f357470f640 1 -- 172.21.15.86:0/69879347 shutdown_connections 2024-03-29T11:29:03.715 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.723+0000 7f357470f640 1 -- 172.21.15.86:0/69879347 wait complete. 2024-03-29T11:29:03.715 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:03.723+0000 7f357470f640 1 librados: shutdown 2024-03-29T11:29:03.730 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:10.537 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:10.697 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.706+0000 7f6a4e250640 10 monclient: get_monmap_and_config 2024-03-29T11:29:10.697 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.706+0000 7f6a4e250640 10 monclient: build_initial_monmap 2024-03-29T11:29:10.697 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.706+0000 7f6a4e250640 10 monclient: monmap: 2024-03-29T11:29:10.697 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:29:10.697 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:10.697 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:29:10.707281+0000 2024-03-29T11:29:10.697 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:29:10.707281+0000 2024-03-29T11:29:10.698 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:10.698 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:10.698 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:29:10.698 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:29:10.698 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:29:10.698 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:10.698 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.707+0000 7f6a4e250640 1 Processor -- start 2024-03-29T11:29:10.698 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.707+0000 7f6a4e250640 1 -- start start 2024-03-29T11:29:10.699 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.707+0000 7f6a4e250640 10 monclient: init 2024-03-29T11:29:10.700 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.708+0000 7f6a4e250640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:10.700 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.708+0000 7f6a4e250640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:29:10.700 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.708+0000 7f6a4e250640 10 monclient(hunting): picked mon.noname-c con 0x7f6a481457f0 addr v1:172.21.15.196:6789/0 2024-03-29T11:29:10.700 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.708+0000 7f6a4e250640 10 monclient(hunting): picked mon.noname-a con 0x7f6a4813f650 addr v1:172.21.15.86:6789/0 2024-03-29T11:29:10.700 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.708+0000 7f6a4e250640 10 monclient(hunting): picked mon.noname-b con 0x7f6a480a9d70 addr v1:172.21.15.86:6790/0 2024-03-29T11:29:10.700 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.708+0000 7f6a4e250640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a480aaef0 con 0x7f6a4813f650 2024-03-29T11:29:10.701 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.708+0000 7f6a4e250640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a48145bd0 con 0x7f6a481457f0 2024-03-29T11:29:10.701 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.708+0000 7f6a4e250640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a48145eb0 con 0x7f6a480a9d70 2024-03-29T11:29:10.701 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.708+0000 7f6a4e250640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:10.701 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.708+0000 7f6a4e250640 10 monclient(hunting): authenticate will time out at 1375.026741s 2024-03-29T11:29:10.701 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.708+0000 7f6a47fff640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f6a480a9d70 0x7f6a4814d8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:49330/0 (socket says 172.21.15.86:49330) 2024-03-29T11:29:10.701 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.709+0000 7f6a47fff640 1 -- 172.21.15.86:0/3742005258 learned_addr learned my addr 172.21.15.86:0/3742005258 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:29:10.704 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.709+0000 7f6a467fc640 1 -- 172.21.15.86:0/3742005258 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2492748104 0 0) 0x7f6a480aaef0 con 0x7f6a4813f650 2024-03-29T11:29:10.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.709+0000 7f6a467fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:10.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.709+0000 7f6a467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:10.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.709+0000 7f6a467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:10.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.709+0000 7f6a467fc640 10 monclient(hunting): my global_id is 4509 2024-03-29T11:29:10.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.710+0000 7f6a467fc640 1 -- 172.21.15.86:0/3742005258 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a34003520 con 0x7f6a4813f650 2024-03-29T11:29:10.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.710+0000 7f6a467fc640 1 -- 172.21.15.86:0/3742005258 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 649952873 0 0) 0x7f6a48145eb0 con 0x7f6a480a9d70 2024-03-29T11:29:10.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.710+0000 7f6a467fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:10.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.710+0000 7f6a467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:10.706 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.710+0000 7f6a467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:10.706 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.710+0000 7f6a467fc640 10 monclient(hunting): my global_id is 4592 2024-03-29T11:29:10.706 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.710+0000 7f6a467fc640 1 -- 172.21.15.86:0/3742005258 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a480aaef0 con 0x7f6a480a9d70 2024-03-29T11:29:10.706 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.710+0000 7f6a467fc640 1 -- 172.21.15.86:0/3742005258 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 845356048 0 0) 0x7f6a48145bd0 con 0x7f6a481457f0 2024-03-29T11:29:10.706 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.710+0000 7f6a467fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:10.706 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.710+0000 7f6a467fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:10.706 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.710+0000 7f6a467fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:10.706 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.710+0000 7f6a467fc640 10 monclient(hunting): my global_id is 4474 2024-03-29T11:29:10.706 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.710+0000 7f6a467fc640 1 -- 172.21.15.86:0/3742005258 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a48145eb0 con 0x7f6a481457f0 2024-03-29T11:29:10.706 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.710+0000 7f6a467fc640 1 -- 172.21.15.86:0/3742005258 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 202109385 0 0) 0x7f6a34003520 con 0x7f6a4813f650 2024-03-29T11:29:10.706 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.710+0000 7f6a467fc640 1 -- 172.21.15.86:0/3742005258 >> v1:172.21.15.86:6790/0 conn(0x7f6a480a9d70 legacy=0x7f6a4814d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:10.706 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.710+0000 7f6a467fc640 1 -- 172.21.15.86:0/3742005258 >> v1:172.21.15.196:6789/0 conn(0x7f6a481457f0 legacy=0x7f6a4813ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:10.706 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.710+0000 7f6a467fc640 10 monclient: _finish_hunting 0 2024-03-29T11:29:10.706 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.711+0000 7f6a467fc640 1 monclient: found mon.noname-a 2024-03-29T11:29:10.707 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.711+0000 7f6a467fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:10.707 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.711+0000 7f6a467fc640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.86:6789/0 2024-03-29T11:29:10.707 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.711+0000 7f6a467fc640 1 -- 172.21.15.86:0/3742005258 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a48146340 con 0x7f6a4813f650 2024-03-29T11:29:10.707 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.711+0000 7f6a467fc640 10 monclient: _finish_auth 0 2024-03-29T11:29:10.707 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.711+0000 7f6a467fc640 10 monclient: _check_auth_tickets 2024-03-29T11:29:10.707 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.711+0000 7f6a467fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:10.707 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.711+0000 7f6a467fc640 1 -- 172.21.15.86:0/3742005258 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f6a30001690 con 0x7f6a4813f650 2024-03-29T11:29:10.707 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.711+0000 7f6a467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:10.707 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.711+0000 7f6a467fc640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:10.707 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.711+0000 7f6a467fc640 10 monclient: dump: 2024-03-29T11:29:10.707 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:29:10.707 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:10.708 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:10.708 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:10.708 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:10.708 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:10.708 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:29:10.708 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:29:10.708 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:29:10.708 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:10.708 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.711+0000 7f6a4e250640 5 monclient: authenticate success, global_id 4509 2024-03-29T11:29:10.708 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.711+0000 7f6a4e250640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:10.708 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.712+0000 7f6a467fc640 1 -- 172.21.15.86:0/3742005258 <== mon.0 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a300018b0 con 0x7f6a4813f650 2024-03-29T11:29:10.708 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.712+0000 7f6a467fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:10.708 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.712+0000 7f6a4e250640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:10.708 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.712+0000 7f6a4e250640 4 set_mon_vals no callback set 2024-03-29T11:29:10.708 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.713+0000 7f6a467fc640 1 -- 172.21.15.86:0/3742005258 <== mon.0 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f6a30001c40 con 0x7f6a4813f650 2024-03-29T11:29:10.709 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.713+0000 7f6a467fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:10.709 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.713+0000 7f6a467fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:10.709 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.713+0000 7f6a467fc640 10 monclient: dump: 2024-03-29T11:29:10.709 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:29:10.709 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:10.709 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:10.709 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:10.709 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:10.709 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:10.709 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:29:10.709 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:29:10.709 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:29:10.709 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:10.709 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.713+0000 7f6a4e250640 10 monclient: shutdown 2024-03-29T11:29:10.709 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.713+0000 7f6a4e250640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:10.710 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.713+0000 7f6a4e250640 1 -- 172.21.15.86:0/3742005258 >> v1:172.21.15.86:6789/0 conn(0x7f6a4813f650 legacy=0x7f6a480b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:10.710 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.713+0000 7f6a4e250640 1 -- 172.21.15.86:0/3742005258 shutdown_connections 2024-03-29T11:29:10.710 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.713+0000 7f6a4e250640 1 -- 172.21.15.86:0/3742005258 >> 172.21.15.86:0/3742005258 conn(0x7f6a48066a00 msgr2=0x7f6a48066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:10.710 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.713+0000 7f6a4e250640 1 -- 172.21.15.86:0/3742005258 shutdown_connections 2024-03-29T11:29:10.710 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.714+0000 7f6a4e250640 1 -- 172.21.15.86:0/3742005258 wait complete. 2024-03-29T11:29:10.717 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.715+0000 7f6a4e250640 10 monclient: build_initial_monmap 2024-03-29T11:29:10.717 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.715+0000 7f6a4e250640 10 monclient: monmap: 2024-03-29T11:29:10.717 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:29:10.717 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:10.717 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:29:10.717 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:29:10.717 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:10.717 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:10.717 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:29:10.717 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:29:10.717 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:29:10.717 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:10.718 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.715+0000 7f6a4e250640 1 librados: starting msgr at 2024-03-29T11:29:10.718 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.715+0000 7f6a4e250640 1 librados: starting objecter 2024-03-29T11:29:10.718 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.715+0000 7f6a4e250640 1 Processor -- start 2024-03-29T11:29:10.718 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.716+0000 7f6a4e250640 1 -- start start 2024-03-29T11:29:10.718 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.716+0000 7f6a4e250640 1 librados: setting wanted keys 2024-03-29T11:29:10.718 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.716+0000 7f6a4e250640 1 librados: calling monclient init 2024-03-29T11:29:10.718 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.716+0000 7f6a4e250640 10 monclient: init 2024-03-29T11:29:10.718 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.716+0000 7f6a4e250640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:10.718 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.716+0000 7f6a4e250640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:29:10.718 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.716+0000 7f6a4e250640 10 monclient(hunting): picked mon.noname-c con 0x7f6a480a9d70 addr v1:172.21.15.86:6790/0 2024-03-29T11:29:10.719 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.716+0000 7f6a4e250640 10 monclient(hunting): picked mon.noname-b con 0x7f6a4813f650 addr v1:172.21.15.196:6789/0 2024-03-29T11:29:10.719 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.716+0000 7f6a4e250640 10 monclient(hunting): picked mon.noname-a con 0x7f6a481457f0 addr v1:172.21.15.86:6789/0 2024-03-29T11:29:10.719 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.716+0000 7f6a4e250640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a48070ce0 con 0x7f6a481457f0 2024-03-29T11:29:10.719 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.716+0000 7f6a4e250640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a48072090 con 0x7f6a4813f650 2024-03-29T11:29:10.719 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.716+0000 7f6a4e250640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a48070f10 con 0x7f6a480a9d70 2024-03-29T11:29:10.719 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.716+0000 7f6a4e250640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:10.719 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.716+0000 7f6a4e250640 10 monclient(hunting): authenticate will time out at 1375.034930s 2024-03-29T11:29:10.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.717+0000 7f6a46ffd640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f6a4813f650 0x7f6a4806f040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:42416/0 (socket says 172.21.15.86:42416) 2024-03-29T11:29:10.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.717+0000 7f6a46ffd640 1 -- 172.21.15.86:0/724477633 learned_addr learned my addr 172.21.15.86:0/724477633 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:29:10.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.717+0000 7f6a3ffff640 10 client.?.objecter ms_handle_connect 0x7f6a4813f650 2024-03-29T11:29:10.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.717+0000 7f6a3ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:10.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.717+0000 7f6a3ffff640 1 -- 172.21.15.86:0/724477633 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2130377229 0 0) 0x7f6a48072090 con 0x7f6a4813f650 2024-03-29T11:29:10.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.717+0000 7f6a3ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:10.720 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.717+0000 7f6a3ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:10.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.717+0000 7f6a3ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:10.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.717+0000 7f6a3ffff640 10 monclient(hunting): my global_id is 4477 2024-03-29T11:29:10.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.717+0000 7f6a3ffff640 1 -- 172.21.15.86:0/724477633 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a28004eb0 con 0x7f6a4813f650 2024-03-29T11:29:10.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.717+0000 7f6a3ffff640 10 client.?.objecter ms_handle_connect 0x7f6a481457f0 2024-03-29T11:29:10.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.717+0000 7f6a3ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:10.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.718+0000 7f6a3ffff640 1 -- 172.21.15.86:0/724477633 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2837528639 0 0) 0x7f6a28004eb0 con 0x7f6a4813f650 2024-03-29T11:29:10.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.718+0000 7f6a3ffff640 1 -- 172.21.15.86:0/724477633 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f6a48072090 con 0x7f6a4813f650 2024-03-29T11:29:10.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.718+0000 7f6a3ffff640 1 -- 172.21.15.86:0/724477633 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f6a30001690 con 0x7f6a4813f650 2024-03-29T11:29:10.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.718+0000 7f6a3ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:10.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.718+0000 7f6a3ffff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:10.721 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.718+0000 7f6a3ffff640 10 monclient(hunting): dump: 2024-03-29T11:29:10.721 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:29:10.722 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:10.722 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:10.722 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:10.722 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:10.722 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:10.722 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:29:10.722 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:29:10.722 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:29:10.722 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:10.722 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.718+0000 7f6a3ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:10.722 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.718+0000 7f6a3ffff640 1 -- 172.21.15.86:0/724477633 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 49679853 0 0) 0x7f6a48072090 con 0x7f6a4813f650 2024-03-29T11:29:10.722 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.718+0000 7f6a3ffff640 1 -- 172.21.15.86:0/724477633 >> v1:172.21.15.86:6790/0 conn(0x7f6a480a9d70 legacy=0x7f6a48071980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:10.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.718+0000 7f6a3ffff640 1 -- 172.21.15.86:0/724477633 >> v1:172.21.15.86:6789/0 conn(0x7f6a481457f0 legacy=0x7f6a4806f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:10.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.718+0000 7f6a3ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:29:10.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.718+0000 7f6a3ffff640 1 monclient: found mon.b 2024-03-29T11:29:10.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.718+0000 7f6a3ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:10.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.718+0000 7f6a3ffff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:29:10.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.718+0000 7f6a3ffff640 1 -- 172.21.15.86:0/724477633 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a4806eb20 con 0x7f6a4813f650 2024-03-29T11:29:10.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.718+0000 7f6a3ffff640 10 monclient: _finish_auth 0 2024-03-29T11:29:10.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.718+0000 7f6a3ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:29:10.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.718+0000 7f6a3ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:10.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.718+0000 7f6a3ffff640 10 client.?.objecter ms_handle_connect 0x7f6a480a9d70 2024-03-29T11:29:10.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.718+0000 7f6a3ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:10.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.718+0000 7f6a4e250640 5 monclient: authenticate success, global_id 4477 2024-03-29T11:29:10.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.718+0000 7f6a4e250640 10 monclient: _renew_subs 2024-03-29T11:29:10.723 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.718+0000 7f6a4e250640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:29:10.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.718+0000 7f6a4e250640 1 -- 172.21.15.86:0/724477633 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a481392b0 con 0x7f6a4813f650 2024-03-29T11:29:10.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.718+0000 7f6a4e250640 10 client.4477.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:10.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.718+0000 7f6a4e250640 10 monclient: _renew_subs 2024-03-29T11:29:10.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.718+0000 7f6a4e250640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:29:10.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.718+0000 7f6a4e250640 1 -- 172.21.15.86:0/724477633 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a48139560 con 0x7f6a4813f650 2024-03-29T11:29:10.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.718+0000 7f6a4e250640 1 librados: init done 2024-03-29T11:29:10.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.719+0000 7f6a3ffff640 1 -- 172.21.15.86:0/724477633 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6a300038a0 con 0x7f6a4813f650 2024-03-29T11:29:10.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.719+0000 7f6a3ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:10.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.719+0000 7f6a3ffff640 1 -- 172.21.15.86:0/724477633 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f6a30003cb0 con 0x7f6a4813f650 2024-03-29T11:29:10.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.719+0000 7f6a3ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:10.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.719+0000 7f6a3ffff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:10.724 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.719+0000 7f6a3ffff640 10 monclient: dump: 2024-03-29T11:29:10.724 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:29:10.724 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:10.724 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:10.725 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:10.725 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:10.725 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:10.725 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:29:10.725 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:29:10.725 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:29:10.725 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:10.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.719+0000 7f6a457fa640 4 set_mon_vals no callback set 2024-03-29T11:29:10.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.720+0000 7f6a4e250640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:10.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.721+0000 7f6a4e250640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:10.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.721+0000 7f6a4e250640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:29:10.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.721+0000 7f6a4e250640 1 -- 172.21.15.86:0/724477633 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a2c005080 con 0x7f6a4813f650 2024-03-29T11:29:10.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.721+0000 7f6a3ffff640 1 -- 172.21.15.86:0/724477633 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f6a30003e10 con 0x7f6a4813f650 2024-03-29T11:29:10.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.722+0000 7f6a3ffff640 1 -- 172.21.15.86:0/724477633 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f6a30084cd0 con 0x7f6a4813f650 2024-03-29T11:29:10.725 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.722+0000 7f6a3ffff640 10 client.4477.objecter ms_dispatch 0x7f6a48000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:10.726 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.722+0000 7f6a3ffff640 3 client.4477.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:10.726 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.722+0000 7f6a3ffff640 3 client.4477.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:10.726 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.722+0000 7f6a3ffff640 20 client.4477.objecter dump_active .. 0 homeless 2024-03-29T11:29:10.726 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.722+0000 7f6a477fe640 10 monclient: get_auth_request con 0x7f6a2806dc40 auth_method 0 2024-03-29T11:29:10.726 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.722+0000 7f6a3ffff640 10 client.4477.objecter ms_handle_connect 0x7f6a2806dc40 2024-03-29T11:29:10.727 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.729+0000 7f6a3ffff640 1 -- 172.21.15.86:0/724477633 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6a300505b0 con 0x7f6a4813f650 2024-03-29T11:29:10.727 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.729+0000 7f6a3ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:10.727 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:10.729+0000 7f6a3ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:11.101 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:11.110+0000 7f6a4e250640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:11.101 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:11.110+0000 7f6a4e250640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:11.101 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:11.110+0000 7f6a4e250640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:29:11.101 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:11.110+0000 7f6a4e250640 1 -- 172.21.15.86:0/724477633 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6a2c005330 con 0x7f6a4813f650 2024-03-29T11:29:11.103 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:11.112+0000 7f6a3ffff640 1 -- 172.21.15.86:0/724477633 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f6a300524e0 con 0x7f6a4813f650 2024-03-29T11:29:11.103 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:11.112+0000 7f6a3ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:11.103 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:11.112+0000 7f6a3ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:11.104 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:29:11.104 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:11.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:11.116+0000 7f6a4e250640 10 librados: watch_flush enter 2024-03-29T11:29:11.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:11.116+0000 7f6a4e250640 10 librados: watch_flush exit 2024-03-29T11:29:11.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:11.116+0000 7f6a4e250640 20 client.4477.objecter shutdown clearing up homeless session... 2024-03-29T11:29:11.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:11.116+0000 7f6a4e250640 10 client.4477.objecter successfully canceled tick 2024-03-29T11:29:11.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:11.116+0000 7f6a4e250640 1 -- 172.21.15.86:0/724477633 >> v1:172.21.15.196:6816/588110318 conn(0x7f6a2806dc40 legacy=0x7f6a28070100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:11.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:11.116+0000 7f6a4e250640 10 monclient: shutdown 2024-03-29T11:29:11.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:11.116+0000 7f6a4e250640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:11.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:11.116+0000 7f6a4e250640 1 -- 172.21.15.86:0/724477633 >> v1:172.21.15.196:6789/0 conn(0x7f6a4813f650 legacy=0x7f6a4806f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:11.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:11.116+0000 7f6a4e250640 1 -- 172.21.15.86:0/724477633 shutdown_connections 2024-03-29T11:29:11.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:11.116+0000 7f6a4e250640 1 -- 172.21.15.86:0/724477633 >> 172.21.15.86:0/724477633 conn(0x7f6a48066a00 msgr2=0x7f6a4813ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:11.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:11.117+0000 7f6a4e250640 1 -- 172.21.15.86:0/724477633 shutdown_connections 2024-03-29T11:29:11.110 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:11.117+0000 7f6a4e250640 1 -- 172.21.15.86:0/724477633 wait complete. 2024-03-29T11:29:11.110 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:11.117+0000 7f6a4e250640 1 librados: shutdown 2024-03-29T11:29:11.123 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:17.931 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:18.083 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.091+0000 7fbe3302a640 10 monclient: get_monmap_and_config 2024-03-29T11:29:18.083 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.091+0000 7fbe3302a640 10 monclient: build_initial_monmap 2024-03-29T11:29:18.083 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.091+0000 7fbe3302a640 10 monclient: monmap: 2024-03-29T11:29:18.083 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:29:18.083 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:18.083 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:29:18.092509+0000 2024-03-29T11:29:18.083 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:29:18.092509+0000 2024-03-29T11:29:18.083 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:18.083 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:18.083 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:29:18.083 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:29:18.083 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:29:18.083 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:18.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.092+0000 7fbe3302a640 1 Processor -- start 2024-03-29T11:29:18.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.092+0000 7fbe3302a640 1 -- start start 2024-03-29T11:29:18.084 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.092+0000 7fbe3302a640 10 monclient: init 2024-03-29T11:29:18.089 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.093+0000 7fbe3302a640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:18.089 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.093+0000 7fbe3302a640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:18.089 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.093+0000 7fbe3302a640 10 monclient(hunting): picked mon.noname-a con 0x7fbe2c0b1f60 addr v1:172.21.15.86:6789/0 2024-03-29T11:29:18.089 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.093+0000 7fbe3302a640 10 monclient(hunting): picked mon.noname-b con 0x7fbe2c071eb0 addr v1:172.21.15.86:6790/0 2024-03-29T11:29:18.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.093+0000 7fbe3302a640 10 monclient(hunting): picked mon.noname-c con 0x7fbe2c143d10 addr v1:172.21.15.196:6789/0 2024-03-29T11:29:18.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.093+0000 7fbe3302a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe2c06ef80 con 0x7fbe2c0b1f60 2024-03-29T11:29:18.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.093+0000 7fbe3302a640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe2c06f260 con 0x7fbe2c143d10 2024-03-29T11:29:18.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.093+0000 7fbe3302a640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe2c0b40c0 con 0x7fbe2c071eb0 2024-03-29T11:29:18.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.093+0000 7fbe3302a640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:18.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.093+0000 7fbe3302a640 10 monclient(hunting): authenticate will time out at 1382.411846s 2024-03-29T11:29:18.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.094+0000 7fbe30dc6640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fbe2c0b1f60 0x7fbe2c143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:45302/0 (socket says 172.21.15.86:45302) 2024-03-29T11:29:18.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.094+0000 7fbe30dc6640 1 -- 172.21.15.86:0/1740689652 learned_addr learned my addr 172.21.15.86:0/1740689652 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:29:18.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.094+0000 7fbe2b7fe640 1 -- 172.21.15.86:0/1740689652 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 594740507 0 0) 0x7fbe2c06ef80 con 0x7fbe2c0b1f60 2024-03-29T11:29:18.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.094+0000 7fbe2b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:18.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.094+0000 7fbe2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:18.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.094+0000 7fbe2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:18.090 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.094+0000 7fbe2b7fe640 10 monclient(hunting): my global_id is 4515 2024-03-29T11:29:18.091 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.094+0000 7fbe2b7fe640 1 -- 172.21.15.86:0/1740689652 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe0c003520 con 0x7fbe2c0b1f60 2024-03-29T11:29:18.091 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.095+0000 7fbe2b7fe640 1 -- 172.21.15.86:0/1740689652 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2748963127 0 0) 0x7fbe2c06f260 con 0x7fbe2c143d10 2024-03-29T11:29:18.091 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.095+0000 7fbe2b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:18.091 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.095+0000 7fbe2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:18.091 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.095+0000 7fbe2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:18.091 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.095+0000 7fbe2b7fe640 10 monclient(hunting): my global_id is 4480 2024-03-29T11:29:18.091 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.095+0000 7fbe2b7fe640 1 -- 172.21.15.86:0/1740689652 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe2c06ef80 con 0x7fbe2c143d10 2024-03-29T11:29:18.091 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.095+0000 7fbe2b7fe640 1 -- 172.21.15.86:0/1740689652 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3552218992 0 0) 0x7fbe2c0b40c0 con 0x7fbe2c071eb0 2024-03-29T11:29:18.091 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.095+0000 7fbe2b7fe640 10 monclient(hunting): _negotiate 2024-03-29T11:29:18.091 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.095+0000 7fbe2b7fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:18.091 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.095+0000 7fbe2b7fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:18.091 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.095+0000 7fbe2b7fe640 10 monclient(hunting): my global_id is 4595 2024-03-29T11:29:18.091 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.095+0000 7fbe2b7fe640 1 -- 172.21.15.86:0/1740689652 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe2c06f260 con 0x7fbe2c071eb0 2024-03-29T11:29:18.091 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.095+0000 7fbe2b7fe640 1 -- 172.21.15.86:0/1740689652 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4006915465 0 0) 0x7fbe0c003520 con 0x7fbe2c0b1f60 2024-03-29T11:29:18.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.095+0000 7fbe2b7fe640 1 -- 172.21.15.86:0/1740689652 >> v1:172.21.15.86:6790/0 conn(0x7fbe2c071eb0 legacy=0x7fbe2c146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:18.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.095+0000 7fbe2b7fe640 1 -- 172.21.15.86:0/1740689652 >> v1:172.21.15.196:6789/0 conn(0x7fbe2c143d10 legacy=0x7fbe2c146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:18.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.095+0000 7fbe2b7fe640 10 monclient: _finish_hunting 0 2024-03-29T11:29:18.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.095+0000 7fbe2b7fe640 1 monclient: found mon.noname-a 2024-03-29T11:29:18.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.095+0000 7fbe2b7fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:18.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.095+0000 7fbe2b7fe640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.86:6789/0 2024-03-29T11:29:18.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.095+0000 7fbe2b7fe640 1 -- 172.21.15.86:0/1740689652 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe2c155d70 con 0x7fbe2c0b1f60 2024-03-29T11:29:18.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.095+0000 7fbe2b7fe640 10 monclient: _finish_auth 0 2024-03-29T11:29:18.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.095+0000 7fbe2b7fe640 10 monclient: _check_auth_tickets 2024-03-29T11:29:18.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.095+0000 7fbe2b7fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:18.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.095+0000 7fbe3302a640 5 monclient: authenticate success, global_id 4515 2024-03-29T11:29:18.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.095+0000 7fbe3302a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:18.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.095+0000 7fbe2b7fe640 1 -- 172.21.15.86:0/1740689652 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fbe24001690 con 0x7fbe2c0b1f60 2024-03-29T11:29:18.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.095+0000 7fbe2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:18.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.095+0000 7fbe2b7fe640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:18.092 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.095+0000 7fbe2b7fe640 10 monclient: dump: 2024-03-29T11:29:18.093 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:29:18.093 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:18.093 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:18.093 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:18.093 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:18.093 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:18.093 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:29:18.093 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:29:18.093 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:29:18.093 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:18.093 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.095+0000 7fbe3302a640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:18.093 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.095+0000 7fbe2b7fe640 1 -- 172.21.15.86:0/1740689652 <== mon.0 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe240018b0 con 0x7fbe2c0b1f60 2024-03-29T11:29:18.093 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.095+0000 7fbe2b7fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:18.093 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.096+0000 7fbe3302a640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:18.093 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.096+0000 7fbe3302a640 4 set_mon_vals no callback set 2024-03-29T11:29:18.094 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.096+0000 7fbe2b7fe640 1 -- 172.21.15.86:0/1740689652 <== mon.0 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fbe24001c40 con 0x7fbe2c0b1f60 2024-03-29T11:29:18.094 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.096+0000 7fbe2b7fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:18.094 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.096+0000 7fbe2b7fe640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:18.094 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.096+0000 7fbe2b7fe640 10 monclient: dump: 2024-03-29T11:29:18.094 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:29:18.094 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:18.094 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:18.094 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:18.094 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:18.094 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:18.094 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:29:18.094 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:29:18.094 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:29:18.095 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:18.095 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.097+0000 7fbe3302a640 10 monclient: shutdown 2024-03-29T11:29:18.095 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.097+0000 7fbe3302a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:18.095 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.097+0000 7fbe3302a640 1 -- 172.21.15.86:0/1740689652 >> v1:172.21.15.86:6789/0 conn(0x7fbe2c0b1f60 legacy=0x7fbe2c143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:18.095 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.097+0000 7fbe3302a640 1 -- 172.21.15.86:0/1740689652 shutdown_connections 2024-03-29T11:29:18.095 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.097+0000 7fbe3302a640 1 -- 172.21.15.86:0/1740689652 >> 172.21.15.86:0/1740689652 conn(0x7fbe2c066a00 msgr2=0x7fbe2c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:18.095 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.097+0000 7fbe3302a640 1 -- 172.21.15.86:0/1740689652 shutdown_connections 2024-03-29T11:29:18.095 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.097+0000 7fbe3302a640 1 -- 172.21.15.86:0/1740689652 wait complete. 2024-03-29T11:29:18.095 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.098+0000 7fbe3302a640 10 monclient: build_initial_monmap 2024-03-29T11:29:18.095 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.098+0000 7fbe3302a640 10 monclient: monmap: 2024-03-29T11:29:18.095 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:29:18.095 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:18.095 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:29:18.095 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:29:18.095 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:18.095 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:18.096 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:29:18.096 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:29:18.096 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:29:18.096 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:18.102 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.099+0000 7fbe3302a640 1 librados: starting msgr at 2024-03-29T11:29:18.102 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.099+0000 7fbe3302a640 1 librados: starting objecter 2024-03-29T11:29:18.102 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.099+0000 7fbe3302a640 1 Processor -- start 2024-03-29T11:29:18.102 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.099+0000 7fbe3302a640 1 -- start start 2024-03-29T11:29:18.102 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.099+0000 7fbe3302a640 1 librados: setting wanted keys 2024-03-29T11:29:18.102 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.099+0000 7fbe3302a640 1 librados: calling monclient init 2024-03-29T11:29:18.102 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.099+0000 7fbe3302a640 10 monclient: init 2024-03-29T11:29:18.102 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.100+0000 7fbe3302a640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:18.103 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.100+0000 7fbe3302a640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:29:18.103 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.100+0000 7fbe30dc6640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7fbe2c0b1f60 0x7fbe2c153810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:41102/0 (socket says 172.21.15.86:41102) 2024-03-29T11:29:18.103 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.100+0000 7fbe3302a640 10 monclient(hunting): picked mon.noname-c con 0x7fbe2c0b1f60 addr v1:172.21.15.86:6790/0 2024-03-29T11:29:18.103 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.100+0000 7fbe3302a640 10 monclient(hunting): picked mon.noname-b con 0x7fbe2c143d10 addr v1:172.21.15.196:6789/0 2024-03-29T11:29:18.103 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.100+0000 7fbe3302a640 10 monclient(hunting): picked mon.noname-a con 0x7fbe2c1372f0 addr v1:172.21.15.86:6789/0 2024-03-29T11:29:18.103 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.100+0000 7fbe3302a640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe2c1420a0 con 0x7fbe2c1372f0 2024-03-29T11:29:18.103 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.100+0000 7fbe3302a640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe2c0710e0 con 0x7fbe2c143d10 2024-03-29T11:29:18.103 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.100+0000 7fbe3302a640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe2c0713c0 con 0x7fbe2c0b1f60 2024-03-29T11:29:18.103 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.100+0000 7fbe3302a640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:18.103 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.100+0000 7fbe3302a640 10 monclient(hunting): authenticate will time out at 1382.418987s 2024-03-29T11:29:18.103 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.101+0000 7fbe30dc6640 1 -- 172.21.15.86:0/1914238575 learned_addr learned my addr 172.21.15.86:0/1914238575 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:29:18.103 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.101+0000 7fbe297fa640 10 client.?.objecter ms_handle_connect 0x7fbe2c0b1f60 2024-03-29T11:29:18.103 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.101+0000 7fbe297fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:18.103 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.101+0000 7fbe297fa640 10 client.?.objecter ms_handle_connect 0x7fbe2c1372f0 2024-03-29T11:29:18.103 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.101+0000 7fbe297fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:18.104 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.101+0000 7fbe297fa640 1 -- 172.21.15.86:0/1914238575 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2365087637 0 0) 0x7fbe2c0713c0 con 0x7fbe2c0b1f60 2024-03-29T11:29:18.104 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.101+0000 7fbe297fa640 10 monclient(hunting): _negotiate 2024-03-29T11:29:18.104 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.101+0000 7fbe297fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:18.104 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.101+0000 7fbe297fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:18.104 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.101+0000 7fbe297fa640 10 monclient(hunting): my global_id is 4598 2024-03-29T11:29:18.104 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.101+0000 7fbe297fa640 1 -- 172.21.15.86:0/1914238575 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe10004eb0 con 0x7fbe2c0b1f60 2024-03-29T11:29:18.104 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.101+0000 7fbe297fa640 10 client.?.objecter ms_handle_connect 0x7fbe2c143d10 2024-03-29T11:29:18.104 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.102+0000 7fbe297fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:18.104 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.102+0000 7fbe297fa640 1 -- 172.21.15.86:0/1914238575 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2923365180 0 0) 0x7fbe2c1420a0 con 0x7fbe2c1372f0 2024-03-29T11:29:18.104 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.102+0000 7fbe297fa640 10 monclient(hunting): _negotiate 2024-03-29T11:29:18.104 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.102+0000 7fbe297fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:18.104 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.102+0000 7fbe297fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:18.104 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.102+0000 7fbe297fa640 10 monclient(hunting): my global_id is 4518 2024-03-29T11:29:18.104 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.102+0000 7fbe297fa640 1 -- 172.21.15.86:0/1914238575 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe2c0713c0 con 0x7fbe2c1372f0 2024-03-29T11:29:18.105 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.102+0000 7fbe297fa640 1 -- 172.21.15.86:0/1914238575 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 790032807 0 0) 0x7fbe10004eb0 con 0x7fbe2c0b1f60 2024-03-29T11:29:18.105 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.102+0000 7fbe297fa640 1 -- 172.21.15.86:0/1914238575 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe2c1420a0 con 0x7fbe2c0b1f60 2024-03-29T11:29:18.105 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.102+0000 7fbe297fa640 1 -- 172.21.15.86:0/1914238575 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fbe24001bc0 con 0x7fbe2c0b1f60 2024-03-29T11:29:18.105 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.102+0000 7fbe297fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:18.105 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.102+0000 7fbe297fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:18.105 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.102+0000 7fbe297fa640 10 monclient(hunting): dump: 2024-03-29T11:29:18.105 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:29:18.105 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:18.105 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:18.105 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:18.105 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:18.105 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:18.105 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:29:18.105 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:29:18.105 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:29:18.106 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:18.106 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.102+0000 7fbe297fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:18.106 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.102+0000 7fbe297fa640 1 -- 172.21.15.86:0/1914238575 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2339631815 0 0) 0x7fbe2c0710e0 con 0x7fbe2c143d10 2024-03-29T11:29:18.106 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.102+0000 7fbe297fa640 10 monclient(hunting): _negotiate 2024-03-29T11:29:18.106 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.102+0000 7fbe297fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:18.106 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.102+0000 7fbe297fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:18.106 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.102+0000 7fbe297fa640 10 monclient(hunting): my global_id is 4483 2024-03-29T11:29:18.106 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.102+0000 7fbe297fa640 1 -- 172.21.15.86:0/1914238575 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe10004eb0 con 0x7fbe2c143d10 2024-03-29T11:29:18.106 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.102+0000 7fbe297fa640 1 -- 172.21.15.86:0/1914238575 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2581684160 0 0) 0x7fbe2c0713c0 con 0x7fbe2c1372f0 2024-03-29T11:29:18.106 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.102+0000 7fbe297fa640 1 -- 172.21.15.86:0/1914238575 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe2c0710e0 con 0x7fbe2c1372f0 2024-03-29T11:29:18.106 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.102+0000 7fbe297fa640 1 -- 172.21.15.86:0/1914238575 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fbe14001b50 con 0x7fbe2c1372f0 2024-03-29T11:29:18.106 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.103+0000 7fbe297fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:18.106 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.103+0000 7fbe297fa640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:18.106 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.103+0000 7fbe297fa640 10 monclient(hunting): dump: 2024-03-29T11:29:18.106 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:29:18.107 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:18.107 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:18.107 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:18.107 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:18.107 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:18.107 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:29:18.107 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:29:18.107 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:29:18.107 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:18.107 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.103+0000 7fbe297fa640 1 -- 172.21.15.86:0/1914238575 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3095134452 0 0) 0x7fbe2c1420a0 con 0x7fbe2c0b1f60 2024-03-29T11:29:18.107 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.103+0000 7fbe297fa640 1 -- 172.21.15.86:0/1914238575 >> v1:172.21.15.196:6789/0 conn(0x7fbe2c143d10 legacy=0x7fbe2c153f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:18.107 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.103+0000 7fbe297fa640 1 -- 172.21.15.86:0/1914238575 >> v1:172.21.15.86:6789/0 conn(0x7fbe2c1372f0 legacy=0x7fbe2c1e6190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:18.107 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.103+0000 7fbe297fa640 10 monclient: _finish_hunting 0 2024-03-29T11:29:18.107 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.103+0000 7fbe297fa640 1 monclient: found mon.c 2024-03-29T11:29:18.107 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.103+0000 7fbe297fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:18.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.103+0000 7fbe297fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:29:18.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.103+0000 7fbe297fa640 1 -- 172.21.15.86:0/1914238575 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe2c06f7c0 con 0x7fbe2c0b1f60 2024-03-29T11:29:18.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.103+0000 7fbe297fa640 10 monclient: _finish_auth 0 2024-03-29T11:29:18.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.103+0000 7fbe297fa640 10 monclient: _check_auth_tickets 2024-03-29T11:29:18.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.103+0000 7fbe297fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:18.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.103+0000 7fbe3302a640 5 monclient: authenticate success, global_id 4598 2024-03-29T11:29:18.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.103+0000 7fbe3302a640 10 monclient: _renew_subs 2024-03-29T11:29:18.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.103+0000 7fbe3302a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:29:18.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.103+0000 7fbe3302a640 1 -- 172.21.15.86:0/1914238575 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe2c142c60 con 0x7fbe2c0b1f60 2024-03-29T11:29:18.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.103+0000 7fbe3302a640 10 client.4598.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:18.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.103+0000 7fbe3302a640 10 monclient: _renew_subs 2024-03-29T11:29:18.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.103+0000 7fbe3302a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:29:18.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.103+0000 7fbe3302a640 1 -- 172.21.15.86:0/1914238575 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe2c1e78b0 con 0x7fbe2c0b1f60 2024-03-29T11:29:18.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.103+0000 7fbe3302a640 1 librados: init done 2024-03-29T11:29:18.108 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.104+0000 7fbe3302a640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:18.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.105+0000 7fbe3302a640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:18.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.105+0000 7fbe3302a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:29:18.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.105+0000 7fbe3302a640 1 -- 172.21.15.86:0/1914238575 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe2c153530 con 0x7fbe2c0b1f60 2024-03-29T11:29:18.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.105+0000 7fbe297fa640 1 -- 172.21.15.86:0/1914238575 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe24001150 con 0x7fbe2c0b1f60 2024-03-29T11:29:18.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.105+0000 7fbe297fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:18.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.105+0000 7fbe297fa640 1 -- 172.21.15.86:0/1914238575 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fbe24001d80 con 0x7fbe2c0b1f60 2024-03-29T11:29:18.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.105+0000 7fbe297fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:18.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.105+0000 7fbe297fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:18.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.105+0000 7fbe2a7fc640 4 set_mon_vals no callback set 2024-03-29T11:29:18.109 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.105+0000 7fbe297fa640 10 monclient: dump: 2024-03-29T11:29:18.109 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:29:18.109 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:18.109 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:18.109 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:18.109 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:18.109 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:18.110 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:29:18.110 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:29:18.110 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:29:18.110 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:18.110 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.107+0000 7fbe297fa640 1 -- 172.21.15.86:0/1914238575 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7fbe2401ac00 con 0x7fbe2c0b1f60 2024-03-29T11:29:18.110 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.108+0000 7fbe2bfff640 10 monclient: get_auth_request con 0x7fbe1006e050 auth_method 0 2024-03-29T11:29:18.110 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.108+0000 7fbe297fa640 1 -- 172.21.15.86:0/1914238575 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7fbe240849a0 con 0x7fbe2c0b1f60 2024-03-29T11:29:18.110 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.108+0000 7fbe297fa640 10 client.4598.objecter ms_dispatch 0x7fbe2c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:18.110 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.108+0000 7fbe297fa640 3 client.4598.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:18.110 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.108+0000 7fbe297fa640 3 client.4598.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:18.110 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.108+0000 7fbe297fa640 20 client.4598.objecter dump_active .. 0 homeless 2024-03-29T11:29:18.110 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.110+0000 7fbe297fa640 10 client.4598.objecter ms_handle_connect 0x7fbe1006e050 2024-03-29T11:29:18.111 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.114+0000 7fbe297fa640 1 -- 172.21.15.86:0/1914238575 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbe240500b0 con 0x7fbe2c0b1f60 2024-03-29T11:29:18.111 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.114+0000 7fbe297fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:18.111 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.114+0000 7fbe297fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:18.472 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.480+0000 7fbe3302a640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:18.472 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.480+0000 7fbe3302a640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:18.472 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.480+0000 7fbe3302a640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:29:18.472 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.480+0000 7fbe3302a640 1 -- 172.21.15.86:0/1914238575 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe2c153530 con 0x7fbe2c0b1f60 2024-03-29T11:29:18.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.482+0000 7fbe297fa640 1 -- 172.21.15.86:0/1914238575 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7fbe2c153530 con 0x7fbe2c0b1f60 2024-03-29T11:29:18.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.482+0000 7fbe297fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:18.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.482+0000 7fbe297fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:18.473 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:29:18.473 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:18.476 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.485+0000 7fbe3302a640 10 librados: watch_flush enter 2024-03-29T11:29:18.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.485+0000 7fbe3302a640 10 librados: watch_flush exit 2024-03-29T11:29:18.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.485+0000 7fbe3302a640 20 client.4598.objecter shutdown clearing up homeless session... 2024-03-29T11:29:18.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.485+0000 7fbe3302a640 10 client.4598.objecter successfully canceled tick 2024-03-29T11:29:18.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.485+0000 7fbe3302a640 1 -- 172.21.15.86:0/1914238575 >> v1:172.21.15.196:6816/588110318 conn(0x7fbe1006e050 legacy=0x7fbe10070510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:18.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.485+0000 7fbe3302a640 10 monclient: shutdown 2024-03-29T11:29:18.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.485+0000 7fbe3302a640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:18.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.485+0000 7fbe3302a640 1 -- 172.21.15.86:0/1914238575 >> v1:172.21.15.86:6790/0 conn(0x7fbe2c0b1f60 legacy=0x7fbe2c153810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:18.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.485+0000 7fbe3302a640 1 -- 172.21.15.86:0/1914238575 shutdown_connections 2024-03-29T11:29:18.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.485+0000 7fbe3302a640 1 -- 172.21.15.86:0/1914238575 >> 172.21.15.86:0/1914238575 conn(0x7fbe2c066a00 msgr2=0x7fbe2c066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:18.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.486+0000 7fbe3302a640 1 -- 172.21.15.86:0/1914238575 shutdown_connections 2024-03-29T11:29:18.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.486+0000 7fbe3302a640 1 -- 172.21.15.86:0/1914238575 wait complete. 2024-03-29T11:29:18.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:18.487+0000 7fbe3302a640 1 librados: shutdown 2024-03-29T11:29:18.492 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:25.300 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:25.450 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.458+0000 7f9755692640 10 monclient: get_monmap_and_config 2024-03-29T11:29:25.450 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.458+0000 7f9755692640 10 monclient: build_initial_monmap 2024-03-29T11:29:25.450 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.458+0000 7f9755692640 10 monclient: monmap: 2024-03-29T11:29:25.450 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:29:25.450 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:25.450 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:29:25.459700+0000 2024-03-29T11:29:25.450 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:29:25.459700+0000 2024-03-29T11:29:25.450 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:25.451 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:25.451 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:29:25.451 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:29:25.451 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:29:25.451 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:25.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.461+0000 7f9755692640 1 Processor -- start 2024-03-29T11:29:25.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.462+0000 7f9755692640 1 -- start start 2024-03-29T11:29:25.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.462+0000 7f9755692640 10 monclient: init 2024-03-29T11:29:25.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.462+0000 7f9755692640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:25.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.462+0000 7f9755692640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:29:25.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.462+0000 7f9755692640 10 monclient(hunting): picked mon.noname-a con 0x7f97500a9750 addr v1:172.21.15.86:6789/0 2024-03-29T11:29:25.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.462+0000 7f9755692640 10 monclient(hunting): picked mon.noname-c con 0x7f97500add40 addr v1:172.21.15.196:6789/0 2024-03-29T11:29:25.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.462+0000 7f9755692640 10 monclient(hunting): picked mon.noname-b con 0x7f97500aad20 addr v1:172.21.15.86:6790/0 2024-03-29T11:29:25.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.462+0000 7f9755692640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9750149870 con 0x7f97500a9750 2024-03-29T11:29:25.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.462+0000 7f9755692640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97500abf80 con 0x7f97500add40 2024-03-29T11:29:25.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.462+0000 7f9755692640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9750149ac0 con 0x7f97500aad20 2024-03-29T11:29:25.458 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.462+0000 7f9755692640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:25.458 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.462+0000 7f9755692640 10 monclient(hunting): authenticate will time out at 1389.780931s 2024-03-29T11:29:25.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.466+0000 7f9754e91640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f97500aad20 0x7f9750151bf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:33839/0 (socket says 172.21.15.86:33839) 2024-03-29T11:29:25.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.466+0000 7f9754e91640 1 -- 172.21.15.86:0/2564698295 learned_addr learned my addr 172.21.15.86:0/2564698295 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:29:25.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.466+0000 7f974effd640 1 -- 172.21.15.86:0/2564698295 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4226970840 0 0) 0x7f9750149ac0 con 0x7f97500aad20 2024-03-29T11:29:25.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.466+0000 7f974effd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:25.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.466+0000 7f974effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:25.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.467+0000 7f974effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:25.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.467+0000 7f974effd640 10 monclient(hunting): my global_id is 4601 2024-03-29T11:29:25.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.467+0000 7f974effd640 1 -- 172.21.15.86:0/2564698295 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9738003520 con 0x7f97500aad20 2024-03-29T11:29:25.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.468+0000 7f974effd640 1 -- 172.21.15.86:0/2564698295 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3839508528 0 0) 0x7f9738003520 con 0x7f97500aad20 2024-03-29T11:29:25.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.468+0000 7f974effd640 1 -- 172.21.15.86:0/2564698295 >> v1:172.21.15.196:6789/0 conn(0x7f97500add40 legacy=0x7f9750143610 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:25.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.468+0000 7f974effd640 1 -- 172.21.15.86:0/2564698295 >> v1:172.21.15.86:6789/0 conn(0x7f97500a9750 legacy=0x7f9750149fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:25.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.468+0000 7f974effd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:25.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.468+0000 7f974effd640 1 monclient: found mon.noname-b 2024-03-29T11:29:25.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.468+0000 7f974effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:25.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.468+0000 7f974effd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:29:25.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.468+0000 7f974effd640 1 -- 172.21.15.86:0/2564698295 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f975014a6f0 con 0x7f97500aad20 2024-03-29T11:29:25.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.468+0000 7f974effd640 10 monclient: _finish_auth 0 2024-03-29T11:29:25.463 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.468+0000 7f974effd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:25.463 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.468+0000 7f974effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:25.463 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.468+0000 7f974effd640 1 -- 172.21.15.86:0/2564698295 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f9744001690 con 0x7f97500aad20 2024-03-29T11:29:25.463 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.468+0000 7f974effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:25.463 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.468+0000 7f974effd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:25.463 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.468+0000 7f974effd640 10 monclient: dump: 2024-03-29T11:29:25.463 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:29:25.463 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:25.463 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:25.463 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:25.463 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:25.463 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:25.463 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:29:25.463 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:29:25.463 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:29:25.464 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:25.464 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.469+0000 7f9755692640 5 monclient: authenticate success, global_id 4601 2024-03-29T11:29:25.464 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.469+0000 7f9755692640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:25.464 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.471+0000 7f974effd640 1 -- 172.21.15.86:0/2564698295 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9744001860 con 0x7f97500aad20 2024-03-29T11:29:25.464 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.471+0000 7f974effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:25.464 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.471+0000 7f974effd640 1 -- 172.21.15.86:0/2564698295 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f9744001cb0 con 0x7f97500aad20 2024-03-29T11:29:25.464 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.471+0000 7f974effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:25.464 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.471+0000 7f974effd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:25.464 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.471+0000 7f974effd640 10 monclient: dump: 2024-03-29T11:29:25.464 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:29:25.464 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:25.464 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:25.464 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:25.464 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:25.465 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:25.465 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:29:25.465 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:29:25.465 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:29:25.465 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:25.465 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.471+0000 7f9755692640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:25.465 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.471+0000 7f9755692640 4 set_mon_vals no callback set 2024-03-29T11:29:25.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.472+0000 7f9755692640 10 monclient: shutdown 2024-03-29T11:29:25.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.472+0000 7f9755692640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:25.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.472+0000 7f9755692640 1 -- 172.21.15.86:0/2564698295 >> v1:172.21.15.86:6790/0 conn(0x7f97500aad20 legacy=0x7f9750151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:25.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.472+0000 7f9755692640 1 -- 172.21.15.86:0/2564698295 shutdown_connections 2024-03-29T11:29:25.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.472+0000 7f9755692640 1 -- 172.21.15.86:0/2564698295 >> 172.21.15.86:0/2564698295 conn(0x7f97500673e0 msgr2=0x7f97500b3820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:25.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.472+0000 7f9755692640 1 -- 172.21.15.86:0/2564698295 shutdown_connections 2024-03-29T11:29:25.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.472+0000 7f9755692640 1 -- 172.21.15.86:0/2564698295 wait complete. 2024-03-29T11:29:25.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.473+0000 7f9755692640 10 monclient: build_initial_monmap 2024-03-29T11:29:25.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.473+0000 7f9755692640 10 monclient: monmap: 2024-03-29T11:29:25.467 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:29:25.467 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:25.467 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:29:25.467 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:29:25.467 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:25.467 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:25.467 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:29:25.467 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:29:25.467 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:29:25.468 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:25.468 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.473+0000 7f9755692640 1 librados: starting msgr at 2024-03-29T11:29:25.468 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.473+0000 7f9755692640 1 librados: starting objecter 2024-03-29T11:29:25.468 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.474+0000 7f9755692640 1 Processor -- start 2024-03-29T11:29:25.468 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.474+0000 7f9755692640 1 -- start start 2024-03-29T11:29:25.468 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.474+0000 7f9755692640 1 librados: setting wanted keys 2024-03-29T11:29:25.468 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.474+0000 7f9755692640 1 librados: calling monclient init 2024-03-29T11:29:25.468 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.474+0000 7f9755692640 10 monclient: init 2024-03-29T11:29:25.468 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.475+0000 7f9755692640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:25.468 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.475+0000 7f9755692640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:25.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.475+0000 7f9755692640 10 monclient(hunting): picked mon.noname-b con 0x7f97500a9750 addr v1:172.21.15.196:6789/0 2024-03-29T11:29:25.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.476+0000 7f9755692640 10 monclient(hunting): picked mon.noname-c con 0x7f97500aad20 addr v1:172.21.15.86:6790/0 2024-03-29T11:29:25.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.476+0000 7f9755692640 10 monclient(hunting): picked mon.noname-a con 0x7f97500add40 addr v1:172.21.15.86:6789/0 2024-03-29T11:29:25.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.476+0000 7f9755692640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97501db6b0 con 0x7f97500add40 2024-03-29T11:29:25.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.476+0000 7f9755692640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97501478b0 con 0x7f97500a9750 2024-03-29T11:29:25.470 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.476+0000 7f9755692640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f97501dc250 con 0x7f97500aad20 2024-03-29T11:29:25.470 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.476+0000 7f9755692640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:25.470 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.476+0000 7f9755692640 10 monclient(hunting): authenticate will time out at 1389.794580s 2024-03-29T11:29:25.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.481+0000 7f974ffff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f97500a9750 0x7f9750147fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:41611/0 (socket says 172.21.15.86:41611) 2024-03-29T11:29:25.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.481+0000 7f974ffff640 1 -- 172.21.15.86:0/2705209654 learned_addr learned my addr 172.21.15.86:0/2705209654 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:29:25.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.481+0000 7f974cff9640 10 client.?.objecter ms_handle_connect 0x7f97500a9750 2024-03-29T11:29:25.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.481+0000 7f974cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:25.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.481+0000 7f974cff9640 10 client.?.objecter ms_handle_connect 0x7f97500aad20 2024-03-29T11:29:25.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.481+0000 7f974cff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:25.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.481+0000 7f974cff9640 1 -- 172.21.15.86:0/2705209654 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3328301228 0 0) 0x7f97501478b0 con 0x7f97500a9750 2024-03-29T11:29:25.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.481+0000 7f974cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:25.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.481+0000 7f974cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:25.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.481+0000 7f974cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:25.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.481+0000 7f974cff9640 10 monclient(hunting): my global_id is 4486 2024-03-29T11:29:25.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.481+0000 7f974cff9640 1 -- 172.21.15.86:0/2705209654 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f973c004eb0 con 0x7f97500a9750 2024-03-29T11:29:25.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.481+0000 7f974cff9640 1 -- 172.21.15.86:0/2705209654 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 588543411 0 0) 0x7f97501dc250 con 0x7f97500aad20 2024-03-29T11:29:25.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.482+0000 7f974cff9640 10 monclient(hunting): _negotiate 2024-03-29T11:29:25.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.482+0000 7f974cff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:25.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.482+0000 7f974cff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:25.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.482+0000 7f974cff9640 10 monclient(hunting): my global_id is 4604 2024-03-29T11:29:25.479 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.482+0000 7f974cff9640 1 -- 172.21.15.86:0/2705209654 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f97501478b0 con 0x7f97500aad20 2024-03-29T11:29:25.479 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.482+0000 7f974cff9640 1 -- 172.21.15.86:0/2705209654 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3568731762 0 0) 0x7f973c004eb0 con 0x7f97500a9750 2024-03-29T11:29:25.479 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.482+0000 7f974cff9640 1 -- 172.21.15.86:0/2705209654 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f97501dc250 con 0x7f97500a9750 2024-03-29T11:29:25.479 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.483+0000 7f974cff9640 1 -- 172.21.15.86:0/2705209654 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3560973974 0 0) 0x7f97501478b0 con 0x7f97500aad20 2024-03-29T11:29:25.479 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.483+0000 7f974cff9640 1 -- 172.21.15.86:0/2705209654 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f973c004eb0 con 0x7f97500aad20 2024-03-29T11:29:25.479 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.483+0000 7f974cff9640 1 -- 172.21.15.86:0/2705209654 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f9748001ce0 con 0x7f97500a9750 2024-03-29T11:29:25.479 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.483+0000 7f974cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:25.479 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.483+0000 7f974cff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:25.479 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.483+0000 7f974cff9640 10 monclient(hunting): dump: 2024-03-29T11:29:25.479 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:29:25.479 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:25.479 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:25.479 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:25.479 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:25.480 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:25.480 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:29:25.480 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:29:25.480 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:29:25.480 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:25.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.483+0000 7f974cff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:25.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.483+0000 7f974cff9640 1 -- 172.21.15.86:0/2705209654 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f9740001730 con 0x7f97500aad20 2024-03-29T11:29:25.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.483+0000 7f974cff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:25.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.483+0000 7f974cff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:25.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.483+0000 7f974cff9640 10 monclient(hunting): dump: 2024-03-29T11:29:25.480 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:29:25.480 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:25.480 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:25.480 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:25.481 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:25.481 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:25.481 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:29:25.481 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:29:25.481 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:29:25.481 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:25.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.483+0000 7f974cff9640 1 -- 172.21.15.86:0/2705209654 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4033174654 0 0) 0x7f973c004eb0 con 0x7f97500aad20 2024-03-29T11:29:25.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.483+0000 7f974cff9640 1 -- 172.21.15.86:0/2705209654 >> v1:172.21.15.196:6789/0 conn(0x7f97500a9750 legacy=0x7f9750147fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:25.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.483+0000 7f974cff9640 1 -- 172.21.15.86:0/2705209654 >> v1:172.21.15.86:6789/0 conn(0x7f97500add40 legacy=0x7f97501dbb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:25.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.483+0000 7f974cff9640 10 monclient: _finish_hunting 0 2024-03-29T11:29:25.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.483+0000 7f974cff9640 1 monclient: found mon.c 2024-03-29T11:29:25.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.483+0000 7f974cff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:25.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.483+0000 7f974cff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:29:25.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.484+0000 7f974cff9640 1 -- 172.21.15.86:0/2705209654 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97501dc3e0 con 0x7f97500aad20 2024-03-29T11:29:25.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.484+0000 7f974cff9640 10 monclient: _finish_auth 0 2024-03-29T11:29:25.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.484+0000 7f974cff9640 10 monclient: _check_auth_tickets 2024-03-29T11:29:25.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.484+0000 7f974cff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:25.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.484+0000 7f974cff9640 1 -- 172.21.15.86:0/2705209654 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9740001890 con 0x7f97500aad20 2024-03-29T11:29:25.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.484+0000 7f974cff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:25.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.484+0000 7f974e7fc640 4 set_mon_vals no callback set 2024-03-29T11:29:25.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.485+0000 7f974cff9640 1 -- 172.21.15.86:0/2705209654 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f97400035f0 con 0x7f97500aad20 2024-03-29T11:29:25.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.485+0000 7f974cff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:25.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.485+0000 7f974cff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:25.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.485+0000 7f974cff9640 10 monclient: dump: 2024-03-29T11:29:25.482 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:29:25.482 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:25.482 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:25.482 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:25.483 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:25.483 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:25.483 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:29:25.483 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:29:25.483 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:29:25.483 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:25.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.485+0000 7f9755692640 5 monclient: authenticate success, global_id 4604 2024-03-29T11:29:25.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.485+0000 7f9755692640 10 monclient: _renew_subs 2024-03-29T11:29:25.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.485+0000 7f9755692640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:29:25.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.485+0000 7f9755692640 1 -- 172.21.15.86:0/2705209654 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97501446f0 con 0x7f97500aad20 2024-03-29T11:29:25.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.485+0000 7f9755692640 10 client.4604.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:25.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.485+0000 7f9755692640 10 monclient: _renew_subs 2024-03-29T11:29:25.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.485+0000 7f9755692640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:29:25.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.485+0000 7f9755692640 1 -- 172.21.15.86:0/2705209654 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f97501e7270 con 0x7f97500aad20 2024-03-29T11:29:25.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.485+0000 7f9755692640 1 librados: init done 2024-03-29T11:29:25.484 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.487+0000 7f9755692640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:25.484 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.487+0000 7f9755692640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:25.484 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.487+0000 7f9755692640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:29:25.484 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.487+0000 7f9755692640 1 -- 172.21.15.86:0/2705209654 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f975004f730 con 0x7f97500aad20 2024-03-29T11:29:25.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.489+0000 7f974cff9640 1 -- 172.21.15.86:0/2705209654 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f9740019640 con 0x7f97500aad20 2024-03-29T11:29:25.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.490+0000 7f974cff9640 1 -- 172.21.15.86:0/2705209654 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f97400836a0 con 0x7f97500aad20 2024-03-29T11:29:25.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.490+0000 7f974cff9640 10 client.4604.objecter ms_dispatch 0x7f9750000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:25.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.490+0000 7f974cff9640 3 client.4604.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:25.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.490+0000 7f974cff9640 3 client.4604.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:25.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.490+0000 7f974cff9640 20 client.4604.objecter dump_active .. 0 homeless 2024-03-29T11:29:25.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.490+0000 7f974ffff640 10 monclient: get_auth_request con 0x7f973c06de00 auth_method 0 2024-03-29T11:29:25.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.494+0000 7f974cff9640 10 client.4604.objecter ms_handle_connect 0x7f973c06de00 2024-03-29T11:29:25.490 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.499+0000 7f974cff9640 1 -- 172.21.15.86:0/2705209654 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f974004edb0 con 0x7f97500aad20 2024-03-29T11:29:25.490 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.499+0000 7f974cff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:25.490 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.499+0000 7f974cff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:25.811 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.819+0000 7f9755692640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:25.811 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.819+0000 7f9755692640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:25.811 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.819+0000 7f9755692640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:29:25.811 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.819+0000 7f9755692640 1 -- 172.21.15.86:0/2705209654 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f97501db860 con 0x7f97500aad20 2024-03-29T11:29:25.811 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.820+0000 7f974cff9640 1 -- 172.21.15.86:0/2705209654 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f97501db860 con 0x7f97500aad20 2024-03-29T11:29:25.811 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.820+0000 7f974cff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:25.812 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.820+0000 7f974cff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:25.813 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:29:25.814 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:25.816 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.824+0000 7f9755692640 10 librados: watch_flush enter 2024-03-29T11:29:25.816 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.824+0000 7f9755692640 10 librados: watch_flush exit 2024-03-29T11:29:25.816 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.824+0000 7f9755692640 20 client.4604.objecter shutdown clearing up homeless session... 2024-03-29T11:29:25.816 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.824+0000 7f9755692640 10 client.4604.objecter successfully canceled tick 2024-03-29T11:29:25.816 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.824+0000 7f9755692640 1 -- 172.21.15.86:0/2705209654 >> v1:172.21.15.196:6816/588110318 conn(0x7f973c06de00 legacy=0x7f973c0702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:25.816 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.824+0000 7f9755692640 10 monclient: shutdown 2024-03-29T11:29:25.816 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.824+0000 7f9755692640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:25.817 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.824+0000 7f9755692640 1 -- 172.21.15.86:0/2705209654 >> v1:172.21.15.86:6790/0 conn(0x7f97500aad20 legacy=0x7f97501471a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:25.817 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.825+0000 7f9755692640 1 -- 172.21.15.86:0/2705209654 shutdown_connections 2024-03-29T11:29:25.817 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.825+0000 7f9755692640 1 -- 172.21.15.86:0/2705209654 >> 172.21.15.86:0/2705209654 conn(0x7f97500673e0 msgr2=0x7f97501404c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:25.817 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.826+0000 7f9755692640 1 -- 172.21.15.86:0/2705209654 shutdown_connections 2024-03-29T11:29:25.817 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.826+0000 7f9755692640 1 -- 172.21.15.86:0/2705209654 wait complete. 2024-03-29T11:29:25.819 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:25.828+0000 7f9755692640 1 librados: shutdown 2024-03-29T11:29:25.836 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:32.643 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:32.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.805+0000 7fe332fc3640 10 monclient: get_monmap_and_config 2024-03-29T11:29:32.796 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.805+0000 7fe332fc3640 10 monclient: build_initial_monmap 2024-03-29T11:29:32.797 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.805+0000 7fe332fc3640 10 monclient: monmap: 2024-03-29T11:29:32.797 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:29:32.797 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:32.797 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:29:32.806181+0000 2024-03-29T11:29:32.797 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:29:32.806181+0000 2024-03-29T11:29:32.797 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:32.797 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:32.797 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:29:32.797 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:29:32.797 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:29:32.797 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:32.798 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.806+0000 7fe332fc3640 1 Processor -- start 2024-03-29T11:29:32.798 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.806+0000 7fe332fc3640 1 -- start start 2024-03-29T11:29:32.798 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.806+0000 7fe332fc3640 10 monclient: init 2024-03-29T11:29:32.799 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.807+0000 7fe332fc3640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:32.799 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.807+0000 7fe332fc3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:32.799 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.807+0000 7fe332fc3640 10 monclient(hunting): picked mon.noname-c con 0x7fe3240389e0 addr v1:172.21.15.196:6789/0 2024-03-29T11:29:32.799 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.807+0000 7fe332fc3640 10 monclient(hunting): picked mon.noname-b con 0x7fe324038dc0 addr v1:172.21.15.86:6790/0 2024-03-29T11:29:32.799 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.807+0000 7fe332fc3640 10 monclient(hunting): picked mon.noname-a con 0x7fe324030660 addr v1:172.21.15.86:6789/0 2024-03-29T11:29:32.799 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.807+0000 7fe332fc3640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3240c54c0 con 0x7fe324030660 2024-03-29T11:29:32.799 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.807+0000 7fe332fc3640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe324036430 con 0x7fe3240389e0 2024-03-29T11:29:32.799 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.807+0000 7fe332fc3640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe324039840 con 0x7fe324038dc0 2024-03-29T11:29:32.799 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.807+0000 7fe332fc3640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:32.799 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.807+0000 7fe332fc3640 10 monclient(hunting): authenticate will time out at 1397.125380s 2024-03-29T11:29:32.800 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.808+0000 7fe331fc1640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fe3240389e0 0x7fe324033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:50276/0 (socket says 172.21.15.86:50276) 2024-03-29T11:29:32.800 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.808+0000 7fe331fc1640 1 -- 172.21.15.86:0/1597837487 learned_addr learned my addr 172.21.15.86:0/1597837487 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:29:32.802 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.810+0000 7fe330fbf640 1 -- 172.21.15.86:0/1597837487 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 623506630 0 0) 0x7fe324036430 con 0x7fe3240389e0 2024-03-29T11:29:32.802 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.810+0000 7fe330fbf640 10 monclient(hunting): _negotiate 2024-03-29T11:29:32.802 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.810+0000 7fe330fbf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:32.802 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.810+0000 7fe330fbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:32.802 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.810+0000 7fe330fbf640 10 monclient(hunting): my global_id is 4489 2024-03-29T11:29:32.802 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.810+0000 7fe330fbf640 1 -- 172.21.15.86:0/1597837487 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe318003520 con 0x7fe3240389e0 2024-03-29T11:29:32.804 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.811+0000 7fe330fbf640 1 -- 172.21.15.86:0/1597837487 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4136345271 0 0) 0x7fe324039840 con 0x7fe324038dc0 2024-03-29T11:29:32.804 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.811+0000 7fe330fbf640 10 monclient(hunting): _negotiate 2024-03-29T11:29:32.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.811+0000 7fe330fbf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:32.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.811+0000 7fe330fbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:32.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.811+0000 7fe330fbf640 10 monclient(hunting): my global_id is 4607 2024-03-29T11:29:32.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.811+0000 7fe330fbf640 1 -- 172.21.15.86:0/1597837487 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe324036430 con 0x7fe324038dc0 2024-03-29T11:29:32.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.811+0000 7fe330fbf640 1 -- 172.21.15.86:0/1597837487 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1705143349 0 0) 0x7fe3240c54c0 con 0x7fe324030660 2024-03-29T11:29:32.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.811+0000 7fe330fbf640 10 monclient(hunting): _negotiate 2024-03-29T11:29:32.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.811+0000 7fe330fbf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:32.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.811+0000 7fe330fbf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:32.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.811+0000 7fe330fbf640 10 monclient(hunting): my global_id is 4521 2024-03-29T11:29:32.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.811+0000 7fe330fbf640 1 -- 172.21.15.86:0/1597837487 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe324039840 con 0x7fe324030660 2024-03-29T11:29:32.805 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.811+0000 7fe330fbf640 1 -- 172.21.15.86:0/1597837487 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3091271354 0 0) 0x7fe324036430 con 0x7fe324038dc0 2024-03-29T11:29:32.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.811+0000 7fe330fbf640 1 -- 172.21.15.86:0/1597837487 >> v1:172.21.15.196:6789/0 conn(0x7fe3240389e0 legacy=0x7fe324033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:32.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.811+0000 7fe330fbf640 1 -- 172.21.15.86:0/1597837487 >> v1:172.21.15.86:6789/0 conn(0x7fe324030660 legacy=0x7fe3240d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:32.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.811+0000 7fe330fbf640 10 monclient: _finish_hunting 0 2024-03-29T11:29:32.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.811+0000 7fe330fbf640 1 monclient: found mon.noname-b 2024-03-29T11:29:32.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.811+0000 7fe330fbf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:32.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.811+0000 7fe330fbf640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:29:32.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.811+0000 7fe330fbf640 1 -- 172.21.15.86:0/1597837487 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe324035690 con 0x7fe324038dc0 2024-03-29T11:29:32.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.811+0000 7fe330fbf640 10 monclient: _finish_auth 0 2024-03-29T11:29:32.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.811+0000 7fe330fbf640 10 monclient: _check_auth_tickets 2024-03-29T11:29:32.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.811+0000 7fe330fbf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:32.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.811+0000 7fe330fbf640 1 -- 172.21.15.86:0/1597837487 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fe328001670 con 0x7fe324038dc0 2024-03-29T11:29:32.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.811+0000 7fe330fbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:32.807 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.811+0000 7fe330fbf640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:32.807 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.811+0000 7fe330fbf640 10 monclient: dump: 2024-03-29T11:29:32.807 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:29:32.807 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:32.807 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:32.807 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:32.807 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:32.807 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:32.807 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:29:32.807 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:29:32.807 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:29:32.807 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:32.808 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.811+0000 7fe332fc3640 5 monclient: authenticate success, global_id 4607 2024-03-29T11:29:32.808 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.811+0000 7fe332fc3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:32.808 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.811+0000 7fe330fbf640 1 -- 172.21.15.86:0/1597837487 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe328001850 con 0x7fe324038dc0 2024-03-29T11:29:32.808 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.811+0000 7fe330fbf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:32.808 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.811+0000 7fe332fc3640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:32.808 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.811+0000 7fe332fc3640 4 set_mon_vals no callback set 2024-03-29T11:29:32.808 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.812+0000 7fe330fbf640 1 -- 172.21.15.86:0/1597837487 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fe328001c60 con 0x7fe324038dc0 2024-03-29T11:29:32.808 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.812+0000 7fe330fbf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:32.808 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.812+0000 7fe330fbf640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:32.808 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.812+0000 7fe330fbf640 10 monclient: dump: 2024-03-29T11:29:32.808 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:29:32.808 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:32.808 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:32.809 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:32.809 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:32.809 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:32.809 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:29:32.809 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:29:32.809 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:29:32.809 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:32.809 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.813+0000 7fe332fc3640 10 monclient: shutdown 2024-03-29T11:29:32.809 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.813+0000 7fe332fc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:32.809 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.813+0000 7fe332fc3640 1 -- 172.21.15.86:0/1597837487 >> v1:172.21.15.86:6790/0 conn(0x7fe324038dc0 legacy=0x7fe3240cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:32.809 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.813+0000 7fe332fc3640 1 -- 172.21.15.86:0/1597837487 shutdown_connections 2024-03-29T11:29:32.809 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.813+0000 7fe332fc3640 1 -- 172.21.15.86:0/1597837487 >> 172.21.15.86:0/1597837487 conn(0x7fe324005c00 msgr2=0x7fe324006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:32.810 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.813+0000 7fe332fc3640 1 -- 172.21.15.86:0/1597837487 shutdown_connections 2024-03-29T11:29:32.810 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.813+0000 7fe332fc3640 1 -- 172.21.15.86:0/1597837487 wait complete. 2024-03-29T11:29:32.812 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.814+0000 7fe332fc3640 10 monclient: build_initial_monmap 2024-03-29T11:29:32.812 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.814+0000 7fe332fc3640 10 monclient: monmap: 2024-03-29T11:29:32.812 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:29:32.812 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:32.812 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:29:32.812 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:29:32.812 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:32.812 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:32.812 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:29:32.813 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:29:32.813 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:29:32.813 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:32.813 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.815+0000 7fe332fc3640 1 librados: starting msgr at 2024-03-29T11:29:32.813 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.815+0000 7fe332fc3640 1 librados: starting objecter 2024-03-29T11:29:32.813 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.815+0000 7fe332fc3640 1 Processor -- start 2024-03-29T11:29:32.813 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.815+0000 7fe332fc3640 1 -- start start 2024-03-29T11:29:32.813 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.815+0000 7fe332fc3640 1 librados: setting wanted keys 2024-03-29T11:29:32.813 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.815+0000 7fe332fc3640 1 librados: calling monclient init 2024-03-29T11:29:32.814 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.815+0000 7fe332fc3640 10 monclient: init 2024-03-29T11:29:32.814 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.816+0000 7fe332fc3640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:32.814 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.816+0000 7fe332fc3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:32.814 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.816+0000 7fe332fc3640 10 monclient(hunting): picked mon.noname-b con 0x7fe324030660 addr v1:172.21.15.196:6789/0 2024-03-29T11:29:32.814 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.816+0000 7fe332fc3640 10 monclient(hunting): picked mon.noname-c con 0x7fe3240389e0 addr v1:172.21.15.86:6790/0 2024-03-29T11:29:32.814 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.817+0000 7fe332fc3640 10 monclient(hunting): picked mon.noname-a con 0x7fe32402d250 addr v1:172.21.15.86:6789/0 2024-03-29T11:29:32.814 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.817+0000 7fe332fc3640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe32416a330 con 0x7fe32402d250 2024-03-29T11:29:32.814 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.817+0000 7fe332fc3640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe32402de10 con 0x7fe324030660 2024-03-29T11:29:32.814 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.817+0000 7fe332fc3640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe324176f40 con 0x7fe3240389e0 2024-03-29T11:29:32.814 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.817+0000 7fe332fc3640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:32.814 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.817+0000 7fe332fc3640 10 monclient(hunting): authenticate will time out at 1397.135301s 2024-03-29T11:29:32.814 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.819+0000 7fe3327c2640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fe32402d250 0x7fe32402d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:59793/0 (socket says 172.21.15.86:59793) 2024-03-29T11:29:32.815 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.819+0000 7fe3327c2640 1 -- 172.21.15.86:0/1816464911 learned_addr learned my addr 172.21.15.86:0/1816464911 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:29:32.816 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.823+0000 7fe3227fc640 10 client.?.objecter ms_handle_connect 0x7fe324030660 2024-03-29T11:29:32.816 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.823+0000 7fe3227fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:32.816 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.823+0000 7fe3227fc640 10 client.?.objecter ms_handle_connect 0x7fe32402d250 2024-03-29T11:29:32.816 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.823+0000 7fe3227fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:32.816 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.823+0000 7fe3227fc640 1 -- 172.21.15.86:0/1816464911 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2003234283 0 0) 0x7fe32402de10 con 0x7fe324030660 2024-03-29T11:29:32.816 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.823+0000 7fe3227fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:32.816 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.823+0000 7fe3227fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:32.816 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.823+0000 7fe3227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:32.816 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.823+0000 7fe3227fc640 10 monclient(hunting): my global_id is 4492 2024-03-29T11:29:32.817 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.823+0000 7fe3227fc640 1 -- 172.21.15.86:0/1816464911 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe310004eb0 con 0x7fe324030660 2024-03-29T11:29:32.817 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.823+0000 7fe3227fc640 1 -- 172.21.15.86:0/1816464911 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 444021184 0 0) 0x7fe32416a330 con 0x7fe32402d250 2024-03-29T11:29:32.817 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.823+0000 7fe3227fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:32.817 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.823+0000 7fe3227fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:32.817 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.823+0000 7fe3227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:32.817 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.823+0000 7fe3227fc640 10 monclient(hunting): my global_id is 4524 2024-03-29T11:29:32.817 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.823+0000 7fe3227fc640 1 -- 172.21.15.86:0/1816464911 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe32402de10 con 0x7fe32402d250 2024-03-29T11:29:32.817 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 10 client.?.objecter ms_handle_connect 0x7fe3240389e0 2024-03-29T11:29:32.817 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:32.817 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 1 -- 172.21.15.86:0/1816464911 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1210217556 0 0) 0x7fe310004eb0 con 0x7fe324030660 2024-03-29T11:29:32.817 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 1 -- 172.21.15.86:0/1816464911 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe32416a330 con 0x7fe324030660 2024-03-29T11:29:32.817 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 1 -- 172.21.15.86:0/1816464911 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fe31c001950 con 0x7fe324030660 2024-03-29T11:29:32.817 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:32.818 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:32.818 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 10 monclient(hunting): dump: 2024-03-29T11:29:32.818 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:29:32.818 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:32.818 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:32.818 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:32.818 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:32.818 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:32.818 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:29:32.818 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:29:32.818 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:29:32.818 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:32.819 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:32.819 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 1 -- 172.21.15.86:0/1816464911 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2652263584 0 0) 0x7fe32402de10 con 0x7fe32402d250 2024-03-29T11:29:32.819 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 1 -- 172.21.15.86:0/1816464911 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fe310004eb0 con 0x7fe32402d250 2024-03-29T11:29:32.819 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 1 -- 172.21.15.86:0/1816464911 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fe32c04fdd0 con 0x7fe32402d250 2024-03-29T11:29:32.819 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:32.819 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:32.819 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 10 monclient(hunting): dump: 2024-03-29T11:29:32.819 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:29:32.819 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:32.820 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:32.820 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:32.820 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:32.820 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:32.820 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:29:32.820 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:29:32.820 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:29:32.820 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:32.820 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 1 -- 172.21.15.86:0/1816464911 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1571882021 0 0) 0x7fe324176f40 con 0x7fe3240389e0 2024-03-29T11:29:32.820 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:32.820 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:32.820 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:32.821 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 10 monclient(hunting): my global_id is 4610 2024-03-29T11:29:32.821 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 1 -- 172.21.15.86:0/1816464911 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe32402de10 con 0x7fe3240389e0 2024-03-29T11:29:32.821 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 1 -- 172.21.15.86:0/1816464911 <== mon.0 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1033823698 0 0) 0x7fe310004eb0 con 0x7fe32402d250 2024-03-29T11:29:32.821 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 1 -- 172.21.15.86:0/1816464911 >> v1:172.21.15.86:6790/0 conn(0x7fe3240389e0 legacy=0x7fe3240c7390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:32.821 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 1 -- 172.21.15.86:0/1816464911 >> v1:172.21.15.196:6789/0 conn(0x7fe324030660 legacy=0x7fe324169ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:32.821 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 10 monclient: _finish_hunting 0 2024-03-29T11:29:32.821 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 1 monclient: found mon.a 2024-03-29T11:29:32.821 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:32.821 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:29:32.821 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 1 -- 172.21.15.86:0/1816464911 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3241780e0 con 0x7fe32402d250 2024-03-29T11:29:32.821 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 10 monclient: _finish_auth 0 2024-03-29T11:29:32.821 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 10 monclient: _check_auth_tickets 2024-03-29T11:29:32.822 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:32.822 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe332fc3640 5 monclient: authenticate success, global_id 4524 2024-03-29T11:29:32.822 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe332fc3640 10 monclient: _renew_subs 2024-03-29T11:29:32.822 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe332fc3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:29:32.822 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe332fc3640 1 -- 172.21.15.86:0/1816464911 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3241770d0 con 0x7fe32402d250 2024-03-29T11:29:32.823 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 1 -- 172.21.15.86:0/1816464911 <== mon.0 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fe32c04f400 con 0x7fe32402d250 2024-03-29T11:29:32.823 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:32.823 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 1 -- 172.21.15.86:0/1816464911 <== mon.0 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fe32c075710 con 0x7fe32402d250 2024-03-29T11:29:32.823 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:32.824 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:32.824 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe3227fc640 10 monclient: dump: 2024-03-29T11:29:32.824 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:29:32.824 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:32.824 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:32.824 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:32.824 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:32.824 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:32.824 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:29:32.824 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:29:32.824 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:29:32.824 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:32.825 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.825+0000 7fe332fc3640 10 client.4524.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:32.825 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.825+0000 7fe332fc3640 10 monclient: _renew_subs 2024-03-29T11:29:32.825 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.825+0000 7fe332fc3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:29:32.825 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.825+0000 7fe332fc3640 1 -- 172.21.15.86:0/1816464911 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fe324177560 con 0x7fe32402d250 2024-03-29T11:29:32.825 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.825+0000 7fe332fc3640 1 librados: init done 2024-03-29T11:29:32.825 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.824+0000 7fe323fff640 4 set_mon_vals no callback set 2024-03-29T11:29:32.825 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.826+0000 7fe332fc3640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:32.825 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.826+0000 7fe3227fc640 1 -- 172.21.15.86:0/1816464911 <== mon.0 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7fe32c067bf0 con 0x7fe32402d250 2024-03-29T11:29:32.825 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.826+0000 7fe332fc3640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:32.825 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.826+0000 7fe332fc3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:29:32.825 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.826+0000 7fe332fc3640 1 -- 172.21.15.86:0/1816464911 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe324007b90 con 0x7fe32402d250 2024-03-29T11:29:32.825 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.827+0000 7fe3227fc640 1 -- 172.21.15.86:0/1816464911 <== mon.0 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7fe32c0ff440 con 0x7fe32402d250 2024-03-29T11:29:32.825 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.827+0000 7fe3227fc640 10 client.4524.objecter ms_dispatch 0x7fe32c000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:32.826 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.827+0000 7fe3227fc640 3 client.4524.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:32.826 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.827+0000 7fe3227fc640 3 client.4524.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:32.826 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.827+0000 7fe3227fc640 20 client.4524.objecter dump_active .. 0 homeless 2024-03-29T11:29:32.826 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.828+0000 7fe331fc1640 10 monclient: get_auth_request con 0x7fe31006dcc0 auth_method 0 2024-03-29T11:29:32.826 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.829+0000 7fe3227fc640 10 client.4524.objecter ms_handle_connect 0x7fe31006dcc0 2024-03-29T11:29:32.826 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.834+0000 7fe3227fc640 1 -- 172.21.15.86:0/1816464911 <== mon.0 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fe32c0cad20 con 0x7fe32402d250 2024-03-29T11:29:32.826 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.834+0000 7fe3227fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:32.827 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:32.834+0000 7fe3227fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:33.156 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:33.164+0000 7fe332fc3640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:33.156 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:33.164+0000 7fe332fc3640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:33.156 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:33.164+0000 7fe332fc3640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:29:33.156 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:33.164+0000 7fe332fc3640 1 -- 172.21.15.86:0/1816464911 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe324005120 con 0x7fe32402d250 2024-03-29T11:29:33.159 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:33.166+0000 7fe3227fc640 1 -- 172.21.15.86:0/1816464911 <== mon.0 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7fe32c067ea0 con 0x7fe32402d250 2024-03-29T11:29:33.159 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:33.166+0000 7fe3227fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:33.159 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:33.166+0000 7fe3227fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:33.159 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:29:33.160 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:33.161 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:33.170+0000 7fe332fc3640 10 librados: watch_flush enter 2024-03-29T11:29:33.161 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:33.170+0000 7fe332fc3640 10 librados: watch_flush exit 2024-03-29T11:29:33.161 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:33.170+0000 7fe332fc3640 20 client.4524.objecter shutdown clearing up homeless session... 2024-03-29T11:29:33.161 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:33.170+0000 7fe332fc3640 10 client.4524.objecter successfully canceled tick 2024-03-29T11:29:33.162 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:33.170+0000 7fe332fc3640 1 -- 172.21.15.86:0/1816464911 >> v1:172.21.15.196:6816/588110318 conn(0x7fe31006dcc0 legacy=0x7fe310070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:33.162 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:33.170+0000 7fe332fc3640 10 monclient: shutdown 2024-03-29T11:29:33.162 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:33.170+0000 7fe332fc3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:33.162 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:33.170+0000 7fe332fc3640 1 -- 172.21.15.86:0/1816464911 >> v1:172.21.15.86:6789/0 conn(0x7fe32402d250 legacy=0x7fe32402d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:33.165 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:33.171+0000 7fe332fc3640 1 -- 172.21.15.86:0/1816464911 shutdown_connections 2024-03-29T11:29:33.165 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:33.171+0000 7fe332fc3640 1 -- 172.21.15.86:0/1816464911 >> 172.21.15.86:0/1816464911 conn(0x7fe324005c00 msgr2=0x7fe3240c3710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:33.166 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:33.171+0000 7fe332fc3640 1 -- 172.21.15.86:0/1816464911 shutdown_connections 2024-03-29T11:29:33.166 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:33.171+0000 7fe332fc3640 1 -- 172.21.15.86:0/1816464911 wait complete. 2024-03-29T11:29:33.166 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:33.171+0000 7fe332fc3640 1 librados: shutdown 2024-03-29T11:29:33.181 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:39.989 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:40.134 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.142+0000 7f46b9dcf640 10 monclient: get_monmap_and_config 2024-03-29T11:29:40.134 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.142+0000 7f46b9dcf640 10 monclient: build_initial_monmap 2024-03-29T11:29:40.134 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.142+0000 7f46b9dcf640 10 monclient: monmap: 2024-03-29T11:29:40.134 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:29:40.134 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:40.135 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:29:40.143543+0000 2024-03-29T11:29:40.135 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:29:40.143543+0000 2024-03-29T11:29:40.135 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:40.135 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:40.135 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:29:40.135 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:29:40.135 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:29:40.135 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:40.137 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.144+0000 7f46b9dcf640 1 Processor -- start 2024-03-29T11:29:40.137 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.144+0000 7f46b9dcf640 1 -- start start 2024-03-29T11:29:40.137 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.144+0000 7f46b9dcf640 10 monclient: init 2024-03-29T11:29:40.137 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.144+0000 7f46b9dcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:40.137 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.144+0000 7f46b9dcf640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:40.137 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.144+0000 7f46b9dcf640 10 monclient(hunting): picked mon.noname-c con 0x7f46b40b1f60 addr v1:172.21.15.196:6789/0 2024-03-29T11:29:40.138 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.144+0000 7f46b9dcf640 10 monclient(hunting): picked mon.noname-b con 0x7f46b4071eb0 addr v1:172.21.15.86:6790/0 2024-03-29T11:29:40.138 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.144+0000 7f46b9dcf640 10 monclient(hunting): picked mon.noname-a con 0x7f46b4143d10 addr v1:172.21.15.86:6789/0 2024-03-29T11:29:40.138 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.144+0000 7f46b9dcf640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b406ef80 con 0x7f46b4143d10 2024-03-29T11:29:40.138 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.144+0000 7f46b9dcf640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b406f260 con 0x7f46b40b1f60 2024-03-29T11:29:40.138 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.144+0000 7f46b9dcf640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b40b40c0 con 0x7f46b4071eb0 2024-03-29T11:29:40.138 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.144+0000 7f46b9dcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:40.138 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.144+0000 7f46b9dcf640 10 monclient(hunting): authenticate will time out at 1404.462916s 2024-03-29T11:29:40.140 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.147+0000 7f46b37fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f46b40b1f60 0x7f46b4143600 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:34514/0 (socket says 172.21.15.86:34514) 2024-03-29T11:29:40.140 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.147+0000 7f46b37fe640 1 -- 172.21.15.86:0/3210602283 learned_addr learned my addr 172.21.15.86:0/3210602283 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:29:40.140 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.148+0000 7f46b27fc640 1 -- 172.21.15.86:0/3210602283 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3331638509 0 0) 0x7f46b406f260 con 0x7f46b40b1f60 2024-03-29T11:29:40.140 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.148+0000 7f46b27fc640 10 monclient(hunting): _negotiate 2024-03-29T11:29:40.140 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.148+0000 7f46b27fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:40.140 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.148+0000 7f46b27fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:40.140 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.148+0000 7f46b27fc640 10 monclient(hunting): my global_id is 4495 2024-03-29T11:29:40.140 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.148+0000 7f46b27fc640 1 -- 172.21.15.86:0/3210602283 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f469c003520 con 0x7f46b40b1f60 2024-03-29T11:29:40.141 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.148+0000 7f46b27fc640 1 -- 172.21.15.86:0/3210602283 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1365054676 0 0) 0x7f469c003520 con 0x7f46b40b1f60 2024-03-29T11:29:40.141 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.148+0000 7f46b27fc640 1 -- 172.21.15.86:0/3210602283 >> v1:172.21.15.86:6790/0 conn(0x7f46b4071eb0 legacy=0x7f46b4146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:40.141 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.148+0000 7f46b27fc640 1 -- 172.21.15.86:0/3210602283 >> v1:172.21.15.86:6789/0 conn(0x7f46b4143d10 legacy=0x7f46b4146b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:40.142 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.151+0000 7f46b27fc640 10 monclient: _finish_hunting 0 2024-03-29T11:29:40.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.151+0000 7f46b27fc640 1 monclient: found mon.noname-c 2024-03-29T11:29:40.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.151+0000 7f46b27fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:40.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.151+0000 7f46b27fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T11:29:40.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.151+0000 7f46b27fc640 1 -- 172.21.15.86:0/3210602283 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b4155d70 con 0x7f46b40b1f60 2024-03-29T11:29:40.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.151+0000 7f46b27fc640 10 monclient: _finish_auth 0 2024-03-29T11:29:40.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.151+0000 7f46b27fc640 10 monclient: _check_auth_tickets 2024-03-29T11:29:40.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.151+0000 7f46b27fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:40.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.151+0000 7f46b9dcf640 5 monclient: authenticate success, global_id 4495 2024-03-29T11:29:40.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.151+0000 7f46b9dcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:40.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.151+0000 7f46b27fc640 1 -- 172.21.15.86:0/3210602283 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f46a8001690 con 0x7f46b40b1f60 2024-03-29T11:29:40.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.151+0000 7f46b27fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:40.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.151+0000 7f46b27fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:40.143 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.151+0000 7f46b27fc640 10 monclient: dump: 2024-03-29T11:29:40.144 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:29:40.144 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:40.144 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:40.144 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:40.144 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:40.144 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:40.144 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:29:40.144 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:29:40.144 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:29:40.144 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:40.144 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.151+0000 7f46b9dcf640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:40.145 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.154+0000 7f46b27fc640 1 -- 172.21.15.86:0/3210602283 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46a8001920 con 0x7f46b40b1f60 2024-03-29T11:29:40.145 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.154+0000 7f46b27fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:40.145 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.154+0000 7f46b9dcf640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:40.145 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.154+0000 7f46b9dcf640 4 set_mon_vals no callback set 2024-03-29T11:29:40.146 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.154+0000 7f46b9dcf640 10 monclient: shutdown 2024-03-29T11:29:40.146 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.154+0000 7f46b9dcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:40.146 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.154+0000 7f46b9dcf640 1 -- 172.21.15.86:0/3210602283 >> v1:172.21.15.196:6789/0 conn(0x7f46b40b1f60 legacy=0x7f46b4143600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:40.146 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.155+0000 7f46b9dcf640 1 -- 172.21.15.86:0/3210602283 shutdown_connections 2024-03-29T11:29:40.146 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.155+0000 7f46b9dcf640 1 -- 172.21.15.86:0/3210602283 >> 172.21.15.86:0/3210602283 conn(0x7f46b4066a00 msgr2=0x7f46b4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:40.147 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.155+0000 7f46b9dcf640 1 -- 172.21.15.86:0/3210602283 shutdown_connections 2024-03-29T11:29:40.147 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.155+0000 7f46b9dcf640 1 -- 172.21.15.86:0/3210602283 wait complete. 2024-03-29T11:29:40.147 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.156+0000 7f46b9dcf640 10 monclient: build_initial_monmap 2024-03-29T11:29:40.147 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.156+0000 7f46b9dcf640 10 monclient: monmap: 2024-03-29T11:29:40.147 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:29:40.147 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:40.148 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:29:40.148 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:29:40.148 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:40.148 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:40.148 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:29:40.148 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:29:40.148 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:29:40.148 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:40.149 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.156+0000 7f46b9dcf640 1 librados: starting msgr at 2024-03-29T11:29:40.149 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.156+0000 7f46b9dcf640 1 librados: starting objecter 2024-03-29T11:29:40.149 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.156+0000 7f46b9dcf640 1 Processor -- start 2024-03-29T11:29:40.149 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.157+0000 7f46b9dcf640 1 -- start start 2024-03-29T11:29:40.149 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.157+0000 7f46b9dcf640 1 librados: setting wanted keys 2024-03-29T11:29:40.149 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.157+0000 7f46b9dcf640 1 librados: calling monclient init 2024-03-29T11:29:40.149 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.157+0000 7f46b9dcf640 10 monclient: init 2024-03-29T11:29:40.149 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.157+0000 7f46b9dcf640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:40.149 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.157+0000 7f46b9dcf640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:40.149 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.157+0000 7f46b9dcf640 10 monclient(hunting): picked mon.noname-a con 0x7f46b40b1f60 addr v1:172.21.15.86:6789/0 2024-03-29T11:29:40.149 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.157+0000 7f46b9dcf640 10 monclient(hunting): picked mon.noname-c con 0x7f46b4143d10 addr v1:172.21.15.86:6790/0 2024-03-29T11:29:40.149 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.157+0000 7f46b9dcf640 10 monclient(hunting): picked mon.noname-b con 0x7f46b4138dc0 addr v1:172.21.15.196:6789/0 2024-03-29T11:29:40.150 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.157+0000 7f46b9dcf640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b4135f40 con 0x7f46b40b1f60 2024-03-29T11:29:40.150 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.157+0000 7f46b9dcf640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b40b7370 con 0x7f46b4138dc0 2024-03-29T11:29:40.150 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.157+0000 7f46b9dcf640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46b413a8a0 con 0x7f46b4143d10 2024-03-29T11:29:40.150 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.157+0000 7f46b9dcf640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:40.150 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.157+0000 7f46b9dcf640 10 monclient(hunting): authenticate will time out at 1404.475903s 2024-03-29T11:29:40.151 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.160+0000 7f46b3fff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f46b4138dc0 0x7f46b41e60b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:35068/0 (socket says 172.21.15.86:35068) 2024-03-29T11:29:40.151 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.160+0000 7f46b3fff640 1 -- 172.21.15.86:0/2420211381 learned_addr learned my addr 172.21.15.86:0/2420211381 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:29:40.153 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.161+0000 7f469bfff640 10 client.?.objecter ms_handle_connect 0x7f46b40b1f60 2024-03-29T11:29:40.153 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.161+0000 7f469bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:40.153 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.161+0000 7f469bfff640 1 -- 172.21.15.86:0/2420211381 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 47114865 0 0) 0x7f46b4155d70 con 0x7f46b40b1f60 2024-03-29T11:29:40.153 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.161+0000 7f469bfff640 10 monclient(hunting): _negotiate 2024-03-29T11:29:40.153 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.161+0000 7f469bfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:40.153 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.161+0000 7f469bfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:40.153 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.161+0000 7f469bfff640 10 monclient(hunting): my global_id is 4527 2024-03-29T11:29:40.153 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.161+0000 7f469bfff640 1 -- 172.21.15.86:0/2420211381 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f46a0004eb0 con 0x7f46b40b1f60 2024-03-29T11:29:40.157 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.162+0000 7f469bfff640 1 -- 172.21.15.86:0/2420211381 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 707701848 0 0) 0x7f46a0004eb0 con 0x7f46b40b1f60 2024-03-29T11:29:40.157 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.162+0000 7f469bfff640 1 -- 172.21.15.86:0/2420211381 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f46b4155d70 con 0x7f46b40b1f60 2024-03-29T11:29:40.157 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.162+0000 7f469bfff640 1 -- 172.21.15.86:0/2420211381 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f46a8001020 con 0x7f46b40b1f60 2024-03-29T11:29:40.157 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.162+0000 7f469bfff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:40.157 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.162+0000 7f469bfff640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:40.157 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.162+0000 7f469bfff640 10 monclient(hunting): dump: 2024-03-29T11:29:40.157 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:29:40.157 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:40.157 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:40.157 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:40.157 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:40.157 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:40.158 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:29:40.158 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:29:40.158 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:29:40.158 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:40.158 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.162+0000 7f469bfff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:40.158 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.162+0000 7f469bfff640 10 client.?.objecter ms_handle_connect 0x7f46b4138dc0 2024-03-29T11:29:40.158 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.162+0000 7f469bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:40.158 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.163+0000 7f469bfff640 10 client.?.objecter ms_handle_connect 0x7f46b4143d10 2024-03-29T11:29:40.158 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.163+0000 7f469bfff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:40.158 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.163+0000 7f469bfff640 1 -- 172.21.15.86:0/2420211381 <== mon.0 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1949870924 0 0) 0x7f46b4155d70 con 0x7f46b40b1f60 2024-03-29T11:29:40.158 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.163+0000 7f469bfff640 1 -- 172.21.15.86:0/2420211381 >> v1:172.21.15.86:6790/0 conn(0x7f46b4143d10 legacy=0x7f46b40b5c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:40.158 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.163+0000 7f469bfff640 1 -- 172.21.15.86:0/2420211381 >> v1:172.21.15.196:6789/0 conn(0x7f46b4138dc0 legacy=0x7f46b41e60b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:40.158 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.163+0000 7f469bfff640 10 monclient: _finish_hunting 0 2024-03-29T11:29:40.158 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.163+0000 7f469bfff640 1 monclient: found mon.a 2024-03-29T11:29:40.159 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.163+0000 7f469bfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:40.159 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.163+0000 7f469bfff640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:29:40.159 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.163+0000 7f469bfff640 1 -- 172.21.15.86:0/2420211381 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46b413ab80 con 0x7f46b40b1f60 2024-03-29T11:29:40.159 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.163+0000 7f469bfff640 10 monclient: _finish_auth 0 2024-03-29T11:29:40.159 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.163+0000 7f469bfff640 10 monclient: _check_auth_tickets 2024-03-29T11:29:40.159 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.163+0000 7f469bfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:40.159 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.163+0000 7f46b9dcf640 5 monclient: authenticate success, global_id 4527 2024-03-29T11:29:40.159 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.163+0000 7f46b9dcf640 10 monclient: _renew_subs 2024-03-29T11:29:40.159 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.163+0000 7f46b9dcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:29:40.159 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.163+0000 7f46b9dcf640 1 -- 172.21.15.86:0/2420211381 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46b40b8450 con 0x7f46b40b1f60 2024-03-29T11:29:40.159 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.163+0000 7f46b9dcf640 10 client.4527.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:40.159 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.163+0000 7f46b9dcf640 10 monclient: _renew_subs 2024-03-29T11:29:40.159 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.163+0000 7f46b9dcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:29:40.159 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.163+0000 7f46b9dcf640 1 -- 172.21.15.86:0/2420211381 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f46b41395b0 con 0x7f46b40b1f60 2024-03-29T11:29:40.159 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.163+0000 7f46b9dcf640 1 librados: init done 2024-03-29T11:29:40.160 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.164+0000 7f469bfff640 1 -- 172.21.15.86:0/2420211381 <== mon.0 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f46a8001b10 con 0x7f46b40b1f60 2024-03-29T11:29:40.160 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.164+0000 7f469bfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:40.160 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.164+0000 7f469bfff640 1 -- 172.21.15.86:0/2420211381 <== mon.0 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f46a8004d90 con 0x7f46b40b1f60 2024-03-29T11:29:40.160 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.164+0000 7f469bfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:40.160 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.164+0000 7f469bfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:29:40.160 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.164+0000 7f469bfff640 10 monclient: dump: 2024-03-29T11:29:40.161 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:29:40.161 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:40.161 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:40.161 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:40.161 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:40.161 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:40.161 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:29:40.161 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:29:40.161 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:29:40.161 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:40.161 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.164+0000 7f46b1ffb640 4 set_mon_vals no callback set 2024-03-29T11:29:40.161 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.165+0000 7f469bfff640 1 -- 172.21.15.86:0/2420211381 <== mon.0 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f46a8004fb0 con 0x7f46b40b1f60 2024-03-29T11:29:40.161 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.166+0000 7f469bfff640 1 -- 172.21.15.86:0/2420211381 <== mon.0 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f46a8084bc0 con 0x7f46b40b1f60 2024-03-29T11:29:40.161 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.166+0000 7f469bfff640 10 client.4527.objecter ms_dispatch 0x7f46b4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:40.161 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.166+0000 7f469bfff640 3 client.4527.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:40.162 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.166+0000 7f469bfff640 3 client.4527.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:40.162 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.166+0000 7f46b2ffd640 10 monclient: get_auth_request con 0x7f46a006dd20 auth_method 0 2024-03-29T11:29:40.162 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.166+0000 7f469bfff640 20 client.4527.objecter dump_active .. 0 homeless 2024-03-29T11:29:40.162 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.166+0000 7f46b9dcf640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:40.162 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.166+0000 7f469bfff640 10 client.4527.objecter ms_handle_connect 0x7f46a006dd20 2024-03-29T11:29:40.162 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.166+0000 7f46b9dcf640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:40.162 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.166+0000 7f46b9dcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:29:40.162 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.166+0000 7f46b9dcf640 1 -- 172.21.15.86:0/2420211381 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4694005080 con 0x7f46b40b1f60 2024-03-29T11:29:40.166 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.174+0000 7f469bfff640 1 -- 172.21.15.86:0/2420211381 <== mon.0 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f46a80504a0 con 0x7f46b40b1f60 2024-03-29T11:29:40.166 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.175+0000 7f469bfff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:40.166 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.175+0000 7f469bfff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:40.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.488+0000 7f46b9dcf640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:40.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.488+0000 7f46b9dcf640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:40.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.488+0000 7f46b9dcf640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:29:40.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.488+0000 7f46b9dcf640 1 -- 172.21.15.86:0/2420211381 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4694005330 con 0x7f46b40b1f60 2024-03-29T11:29:40.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.489+0000 7f469bfff640 1 -- 172.21.15.86:0/2420211381 <== mon.0 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f46a80523d0 con 0x7f46b40b1f60 2024-03-29T11:29:40.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.489+0000 7f469bfff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:40.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.489+0000 7f469bfff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:40.481 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:29:40.481 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:40.484 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.492+0000 7f46b9dcf640 10 librados: watch_flush enter 2024-03-29T11:29:40.484 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.492+0000 7f46b9dcf640 10 librados: watch_flush exit 2024-03-29T11:29:40.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.492+0000 7f46b9dcf640 20 client.4527.objecter shutdown clearing up homeless session... 2024-03-29T11:29:40.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.492+0000 7f46b9dcf640 10 client.4527.objecter successfully canceled tick 2024-03-29T11:29:40.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.493+0000 7f46b9dcf640 1 -- 172.21.15.86:0/2420211381 >> v1:172.21.15.196:6816/588110318 conn(0x7f46a006dd20 legacy=0x7f46a00701e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:40.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.493+0000 7f46b9dcf640 10 monclient: shutdown 2024-03-29T11:29:40.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.493+0000 7f46b9dcf640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:40.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.493+0000 7f46b9dcf640 1 -- 172.21.15.86:0/2420211381 >> v1:172.21.15.86:6789/0 conn(0x7f46b40b1f60 legacy=0x7f46b4136450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:40.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.494+0000 7f46b9dcf640 1 -- 172.21.15.86:0/2420211381 shutdown_connections 2024-03-29T11:29:40.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.494+0000 7f46b9dcf640 1 -- 172.21.15.86:0/2420211381 >> 172.21.15.86:0/2420211381 conn(0x7f46b4066a00 msgr2=0x7f46b4066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:40.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.494+0000 7f46b9dcf640 1 -- 172.21.15.86:0/2420211381 shutdown_connections 2024-03-29T11:29:40.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.494+0000 7f46b9dcf640 1 -- 172.21.15.86:0/2420211381 wait complete. 2024-03-29T11:29:40.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:40.495+0000 7f46b9dcf640 1 librados: shutdown 2024-03-29T11:29:40.504 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:47.311 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:47.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.466+0000 7f81d4339640 10 monclient: get_monmap_and_config 2024-03-29T11:29:47.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.466+0000 7f81d4339640 10 monclient: build_initial_monmap 2024-03-29T11:29:47.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.466+0000 7f81d4339640 10 monclient: monmap: 2024-03-29T11:29:47.460 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:29:47.460 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:47.460 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:29:47.467404+0000 2024-03-29T11:29:47.460 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:29:47.467404+0000 2024-03-29T11:29:47.460 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:47.460 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:47.460 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:29:47.460 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:29:47.460 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:29:47.461 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:47.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.468+0000 7f81d4339640 1 Processor -- start 2024-03-29T11:29:47.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.468+0000 7f81d4339640 1 -- start start 2024-03-29T11:29:47.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.468+0000 7f81d4339640 10 monclient: init 2024-03-29T11:29:47.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.469+0000 7f81d4339640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:47.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.469+0000 7f81d4339640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:29:47.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.469+0000 7f81d4339640 10 monclient(hunting): picked mon.noname-a con 0x7f81cc0aa2a0 addr v1:172.21.15.86:6789/0 2024-03-29T11:29:47.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.469+0000 7f81d4339640 10 monclient(hunting): picked mon.noname-b con 0x7f81cc141f20 addr v1:172.21.15.86:6790/0 2024-03-29T11:29:47.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.469+0000 7f81d4339640 10 monclient(hunting): picked mon.noname-c con 0x7f81cc13b3c0 addr v1:172.21.15.196:6789/0 2024-03-29T11:29:47.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.469+0000 7f81d4339640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81cc13e5c0 con 0x7f81cc0aa2a0 2024-03-29T11:29:47.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.469+0000 7f81d4339640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81cc0ab190 con 0x7f81cc13b3c0 2024-03-29T11:29:47.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.469+0000 7f81d4339640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81cc0ab470 con 0x7f81cc141f20 2024-03-29T11:29:47.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.469+0000 7f81d4339640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:47.462 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.469+0000 7f81d4339640 10 monclient(hunting): authenticate will time out at 1411.787970s 2024-03-29T11:29:47.465 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.473+0000 7f81d28d6640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f81cc13b3c0 0x7f81cc13b810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:59552/0 (socket says 172.21.15.86:59552) 2024-03-29T11:29:47.465 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.473+0000 7f81d28d6640 1 -- 172.21.15.86:0/1176268468 learned_addr learned my addr 172.21.15.86:0/1176268468 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:29:47.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.474+0000 7f81d10d3640 1 -- 172.21.15.86:0/1176268468 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 27270038 0 0) 0x7f81cc0ab470 con 0x7f81cc141f20 2024-03-29T11:29:47.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.474+0000 7f81d10d3640 10 monclient(hunting): _negotiate 2024-03-29T11:29:47.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.474+0000 7f81d10d3640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:47.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.474+0000 7f81d10d3640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:47.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.474+0000 7f81d10d3640 10 monclient(hunting): my global_id is 4619 2024-03-29T11:29:47.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.474+0000 7f81d10d3640 1 -- 172.21.15.86:0/1176268468 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81b8003440 con 0x7f81cc141f20 2024-03-29T11:29:47.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.474+0000 7f81d10d3640 1 -- 172.21.15.86:0/1176268468 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 317853738 0 0) 0x7f81cc13e5c0 con 0x7f81cc0aa2a0 2024-03-29T11:29:47.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.474+0000 7f81d10d3640 10 monclient(hunting): _negotiate 2024-03-29T11:29:47.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.474+0000 7f81d10d3640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:47.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.474+0000 7f81d10d3640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:47.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.474+0000 7f81d10d3640 10 monclient(hunting): my global_id is 4530 2024-03-29T11:29:47.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.474+0000 7f81d10d3640 1 -- 172.21.15.86:0/1176268468 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81cc0ab470 con 0x7f81cc0aa2a0 2024-03-29T11:29:47.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.474+0000 7f81d10d3640 1 -- 172.21.15.86:0/1176268468 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2340146674 0 0) 0x7f81cc0ab470 con 0x7f81cc0aa2a0 2024-03-29T11:29:47.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.474+0000 7f81d10d3640 1 -- 172.21.15.86:0/1176268468 >> v1:172.21.15.86:6790/0 conn(0x7f81cc141f20 legacy=0x7f81cc142340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:47.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.474+0000 7f81d10d3640 1 -- 172.21.15.86:0/1176268468 >> v1:172.21.15.196:6789/0 conn(0x7f81cc13b3c0 legacy=0x7f81cc13b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:29:47.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.474+0000 7f81d10d3640 10 monclient: _finish_hunting 0 2024-03-29T11:29:47.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.474+0000 7f81d10d3640 1 monclient: found mon.noname-a 2024-03-29T11:29:47.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.474+0000 7f81d10d3640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:47.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.474+0000 7f81d10d3640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.86:6789/0 2024-03-29T11:29:47.468 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.474+0000 7f81d10d3640 1 -- 172.21.15.86:0/1176268468 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81cc142a50 con 0x7f81cc0aa2a0 2024-03-29T11:29:47.468 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.474+0000 7f81d10d3640 10 monclient: _finish_auth 0 2024-03-29T11:29:47.468 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.474+0000 7f81d10d3640 10 monclient: _check_auth_tickets 2024-03-29T11:29:47.468 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.474+0000 7f81d10d3640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:47.468 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.474+0000 7f81d4339640 5 monclient: authenticate success, global_id 4530 2024-03-29T11:29:47.468 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.474+0000 7f81d4339640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:47.468 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.474+0000 7f81d10d3640 1 -- 172.21.15.86:0/1176268468 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f81bc001670 con 0x7f81cc0aa2a0 2024-03-29T11:29:47.468 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.474+0000 7f81d10d3640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:47.468 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.474+0000 7f81d10d3640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:29:47.468 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.474+0000 7f81d10d3640 10 monclient: dump: 2024-03-29T11:29:47.468 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:29:47.468 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:47.468 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:47.468 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:47.469 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:47.469 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:47.469 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:29:47.469 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:29:47.469 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:29:47.469 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:47.469 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.475+0000 7f81d4339640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:47.470 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.475+0000 7f81d10d3640 1 -- 172.21.15.86:0/1176268468 <== mon.0 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81bc001890 con 0x7f81cc0aa2a0 2024-03-29T11:29:47.470 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.475+0000 7f81d10d3640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:47.470 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.475+0000 7f81d4339640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:47.470 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.475+0000 7f81d4339640 4 set_mon_vals no callback set 2024-03-29T11:29:47.470 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.476+0000 7f81d4339640 10 monclient: shutdown 2024-03-29T11:29:47.471 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.476+0000 7f81d4339640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:47.471 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.476+0000 7f81d4339640 1 -- 172.21.15.86:0/1176268468 >> v1:172.21.15.86:6789/0 conn(0x7f81cc0aa2a0 legacy=0x7f81cc0ad270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:47.471 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.477+0000 7f81d4339640 1 -- 172.21.15.86:0/1176268468 shutdown_connections 2024-03-29T11:29:47.471 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.477+0000 7f81d4339640 1 -- 172.21.15.86:0/1176268468 >> 172.21.15.86:0/1176268468 conn(0x7f81cc0b35e0 msgr2=0x7f81cc0b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:47.471 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.477+0000 7f81d4339640 1 -- 172.21.15.86:0/1176268468 shutdown_connections 2024-03-29T11:29:47.471 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.477+0000 7f81d4339640 1 -- 172.21.15.86:0/1176268468 wait complete. 2024-03-29T11:29:47.471 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.478+0000 7f81d4339640 10 monclient: build_initial_monmap 2024-03-29T11:29:47.471 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.478+0000 7f81d4339640 10 monclient: monmap: 2024-03-29T11:29:47.471 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:29:47.471 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:47.471 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:29:47.471 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:29:47.471 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:47.471 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:47.471 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:29:47.471 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:29:47.472 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:29:47.472 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:47.472 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.478+0000 7f81d4339640 1 librados: starting msgr at 2024-03-29T11:29:47.472 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.478+0000 7f81d4339640 1 librados: starting objecter 2024-03-29T11:29:47.472 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.479+0000 7f81d4339640 1 Processor -- start 2024-03-29T11:29:47.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.479+0000 7f81d4339640 1 -- start start 2024-03-29T11:29:47.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.479+0000 7f81d4339640 1 librados: setting wanted keys 2024-03-29T11:29:47.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.479+0000 7f81d4339640 1 librados: calling monclient init 2024-03-29T11:29:47.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.479+0000 7f81d4339640 10 monclient: init 2024-03-29T11:29:47.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.480+0000 7f81d4339640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:47.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.480+0000 7f81d4339640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:29:47.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.480+0000 7f81d4339640 10 monclient(hunting): picked mon.noname-a con 0x7f81cc0aa2a0 addr v1:172.21.15.86:6789/0 2024-03-29T11:29:47.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.480+0000 7f81d4339640 10 monclient(hunting): picked mon.noname-b con 0x7f81cc13b3c0 addr v1:172.21.15.196:6789/0 2024-03-29T11:29:47.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.480+0000 7f81d4339640 10 monclient(hunting): picked mon.noname-c con 0x7f81cc141f20 addr v1:172.21.15.86:6790/0 2024-03-29T11:29:47.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.480+0000 7f81d4339640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81cc1c9860 con 0x7f81cc0aa2a0 2024-03-29T11:29:47.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.480+0000 7f81d4339640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81cc1c9b40 con 0x7f81cc13b3c0 2024-03-29T11:29:47.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.480+0000 7f81d4339640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81cc1ca2b0 con 0x7f81cc141f20 2024-03-29T11:29:47.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.480+0000 7f81d4339640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:47.473 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.480+0000 7f81d4339640 10 monclient(hunting): authenticate will time out at 1411.798478s 2024-03-29T11:29:47.474 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.481+0000 7f81d18d4640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f81cc13b3c0 0x7f81cc1d60e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:32916/0 (socket says 172.21.15.86:32916) 2024-03-29T11:29:47.474 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.481+0000 7f81d18d4640 1 -- 172.21.15.86:0/514939716 learned_addr learned my addr 172.21.15.86:0/514939716 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:29:47.474 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.482+0000 7f81c2ffd640 10 client.?.objecter ms_handle_connect 0x7f81cc13b3c0 2024-03-29T11:29:47.474 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.482+0000 7f81c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:47.476 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.484+0000 7f81c2ffd640 1 -- 172.21.15.86:0/514939716 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3727907108 0 0) 0x7f81cc1c9b40 con 0x7f81cc13b3c0 2024-03-29T11:29:47.476 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.484+0000 7f81c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:47.476 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.484+0000 7f81c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:47.476 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.484+0000 7f81c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:47.476 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.484+0000 7f81c2ffd640 10 monclient(hunting): my global_id is 4501 2024-03-29T11:29:47.476 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.484+0000 7f81c2ffd640 1 -- 172.21.15.86:0/514939716 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81b0004eb0 con 0x7f81cc13b3c0 2024-03-29T11:29:47.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.485+0000 7f81c2ffd640 10 client.?.objecter ms_handle_connect 0x7f81cc141f20 2024-03-29T11:29:47.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.485+0000 7f81c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:47.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.485+0000 7f81c2ffd640 1 -- 172.21.15.86:0/514939716 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 220065695 0 0) 0x7f81cc1ca2b0 con 0x7f81cc141f20 2024-03-29T11:29:47.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.485+0000 7f81c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:47.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.485+0000 7f81c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:47.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.485+0000 7f81c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:47.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.485+0000 7f81c2ffd640 10 monclient(hunting): my global_id is 4622 2024-03-29T11:29:47.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.485+0000 7f81c2ffd640 1 -- 172.21.15.86:0/514939716 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81cc1c9b40 con 0x7f81cc141f20 2024-03-29T11:29:47.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.486+0000 7f81c2ffd640 10 client.?.objecter ms_handle_connect 0x7f81cc0aa2a0 2024-03-29T11:29:47.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.486+0000 7f81c2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:47.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.486+0000 7f81c2ffd640 1 -- 172.21.15.86:0/514939716 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2300603656 0 0) 0x7f81cc1c9b40 con 0x7f81cc141f20 2024-03-29T11:29:47.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.486+0000 7f81c2ffd640 1 -- 172.21.15.86:0/514939716 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81cc1ca2b0 con 0x7f81cc141f20 2024-03-29T11:29:47.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.486+0000 7f81c2ffd640 1 -- 172.21.15.86:0/514939716 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f81c8001510 con 0x7f81cc141f20 2024-03-29T11:29:47.479 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.486+0000 7f81c2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:47.479 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.486+0000 7f81c2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:47.479 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.486+0000 7f81c2ffd640 10 monclient(hunting): dump: 2024-03-29T11:29:47.479 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:29:47.479 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:47.479 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:47.479 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:47.479 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:47.479 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:47.479 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:29:47.479 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:29:47.479 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:29:47.479 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:47.479 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.486+0000 7f81c2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:47.479 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.486+0000 7f81c2ffd640 1 -- 172.21.15.86:0/514939716 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2186019585 0 0) 0x7f81cc1c9860 con 0x7f81cc0aa2a0 2024-03-29T11:29:47.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.486+0000 7f81c2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:47.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.486+0000 7f81c2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:47.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.486+0000 7f81c2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:47.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.486+0000 7f81c2ffd640 10 monclient(hunting): my global_id is 4533 2024-03-29T11:29:47.480 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.486+0000 7f81c2ffd640 1 -- 172.21.15.86:0/514939716 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f81cc1c9b40 con 0x7f81cc0aa2a0 2024-03-29T11:29:47.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.486+0000 7f81c2ffd640 1 -- 172.21.15.86:0/514939716 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1214716191 0 0) 0x7f81cc1c9b40 con 0x7f81cc0aa2a0 2024-03-29T11:29:47.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.486+0000 7f81c2ffd640 1 -- 172.21.15.86:0/514939716 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81b0005b00 con 0x7f81cc0aa2a0 2024-03-29T11:29:47.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.486+0000 7f81c2ffd640 1 -- 172.21.15.86:0/514939716 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 547795493 0 0) 0x7f81cc1ca2b0 con 0x7f81cc141f20 2024-03-29T11:29:47.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.486+0000 7f81c2ffd640 1 -- 172.21.15.86:0/514939716 >> v1:172.21.15.196:6789/0 conn(0x7f81cc13b3c0 legacy=0x7f81cc1d60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:47.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.486+0000 7f81c2ffd640 1 -- 172.21.15.86:0/514939716 >> v1:172.21.15.86:6789/0 conn(0x7f81cc0aa2a0 legacy=0x7f81cc132810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:47.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.486+0000 7f81c2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:47.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.486+0000 7f81c2ffd640 1 monclient: found mon.c 2024-03-29T11:29:47.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.486+0000 7f81c2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:47.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.486+0000 7f81c2ffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:29:47.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.486+0000 7f81c2ffd640 1 -- 172.21.15.86:0/514939716 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81cc1c9f60 con 0x7f81cc141f20 2024-03-29T11:29:47.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.486+0000 7f81c2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:29:47.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.486+0000 7f81c2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:47.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.486+0000 7f81c2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:47.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.487+0000 7f81c2ffd640 1 -- 172.21.15.86:0/514939716 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f81c8001a40 con 0x7f81cc141f20 2024-03-29T11:29:47.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.487+0000 7f81c2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:47.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.487+0000 7f81c2ffd640 1 -- 172.21.15.86:0/514939716 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f81c8004a90 con 0x7f81cc141f20 2024-03-29T11:29:47.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.487+0000 7f81c2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:47.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.487+0000 7f81c2ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:29:47.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.487+0000 7f81c2ffd640 10 monclient: dump: 2024-03-29T11:29:47.483 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:29:47.483 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:47.483 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:47.483 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:47.483 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:47.483 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:47.483 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:29:47.483 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:29:47.484 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:29:47.484 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:47.484 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.487+0000 7f81c3fff640 4 set_mon_vals no callback set 2024-03-29T11:29:47.484 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.488+0000 7f81d4339640 5 monclient: authenticate success, global_id 4622 2024-03-29T11:29:47.484 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.488+0000 7f81d4339640 10 monclient: _renew_subs 2024-03-29T11:29:47.484 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.488+0000 7f81d4339640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:29:47.484 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.488+0000 7f81d4339640 1 -- 172.21.15.86:0/514939716 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81cc132f20 con 0x7f81cc141f20 2024-03-29T11:29:47.484 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.488+0000 7f81d4339640 10 client.4622.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:47.484 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.488+0000 7f81d4339640 10 monclient: _renew_subs 2024-03-29T11:29:47.484 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.488+0000 7f81d4339640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:29:47.484 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.488+0000 7f81d4339640 1 -- 172.21.15.86:0/514939716 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f81cc133440 con 0x7f81cc141f20 2024-03-29T11:29:47.484 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.488+0000 7f81d4339640 1 librados: init done 2024-03-29T11:29:47.484 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.490+0000 7f81d4339640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:47.484 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.490+0000 7f81d4339640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:47.484 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.490+0000 7f81d4339640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:29:47.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.490+0000 7f81d4339640 1 -- 172.21.15.86:0/514939716 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81cc075f10 con 0x7f81cc141f20 2024-03-29T11:29:47.485 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.493+0000 7f81c2ffd640 1 -- 172.21.15.86:0/514939716 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f81c801aa60 con 0x7f81cc141f20 2024-03-29T11:29:47.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.494+0000 7f81d20d5640 10 monclient: get_auth_request con 0x7f81b006e0b0 auth_method 0 2024-03-29T11:29:47.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.494+0000 7f81c2ffd640 1 -- 172.21.15.86:0/514939716 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f81c8054600 con 0x7f81cc141f20 2024-03-29T11:29:47.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.494+0000 7f81c2ffd640 10 client.4622.objecter ms_dispatch 0x7f81cc000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:47.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.494+0000 7f81c2ffd640 3 client.4622.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:47.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.494+0000 7f81c2ffd640 3 client.4622.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:47.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.494+0000 7f81c2ffd640 20 client.4622.objecter dump_active .. 0 homeless 2024-03-29T11:29:47.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.494+0000 7f81c2ffd640 10 client.4622.objecter ms_handle_connect 0x7f81b006e0b0 2024-03-29T11:29:47.490 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.498+0000 7f81c2ffd640 1 -- 172.21.15.86:0/514939716 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f81c8050110 con 0x7f81cc141f20 2024-03-29T11:29:47.490 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.498+0000 7f81c2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:47.490 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.498+0000 7f81c2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:47.820 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.828+0000 7f81d4339640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:47.820 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.828+0000 7f81d4339640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:47.820 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.828+0000 7f81d4339640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:29:47.820 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.828+0000 7f81d4339640 1 -- 172.21.15.86:0/514939716 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f81cc1ca0f0 con 0x7f81cc141f20 2024-03-29T11:29:47.821 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.829+0000 7f81c2ffd640 1 -- 172.21.15.86:0/514939716 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f81c8052040 con 0x7f81cc141f20 2024-03-29T11:29:47.821 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.829+0000 7f81c2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:47.821 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.829+0000 7f81c2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:47.822 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:29:47.822 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:47.825 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.833+0000 7f81d4339640 10 librados: watch_flush enter 2024-03-29T11:29:47.825 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.833+0000 7f81d4339640 10 librados: watch_flush exit 2024-03-29T11:29:47.825 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.833+0000 7f81d4339640 20 client.4622.objecter shutdown clearing up homeless session... 2024-03-29T11:29:47.825 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.833+0000 7f81d4339640 10 client.4622.objecter successfully canceled tick 2024-03-29T11:29:47.825 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.833+0000 7f81d4339640 1 -- 172.21.15.86:0/514939716 >> v1:172.21.15.196:6816/588110318 conn(0x7f81b006e0b0 legacy=0x7f81b0070570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:47.825 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.833+0000 7f81d4339640 10 monclient: shutdown 2024-03-29T11:29:47.825 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.833+0000 7f81d4339640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:47.825 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.833+0000 7f81d4339640 1 -- 172.21.15.86:0/514939716 >> v1:172.21.15.86:6790/0 conn(0x7f81cc141f20 legacy=0x7f81cc1de420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:47.826 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.833+0000 7f81d4339640 1 -- 172.21.15.86:0/514939716 shutdown_connections 2024-03-29T11:29:47.826 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.833+0000 7f81d4339640 1 -- 172.21.15.86:0/514939716 >> 172.21.15.86:0/514939716 conn(0x7f81cc0b35e0 msgr2=0x7f81cc0a97a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:47.826 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.833+0000 7f81d4339640 1 -- 172.21.15.86:0/514939716 shutdown_connections 2024-03-29T11:29:47.826 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.833+0000 7f81d4339640 1 -- 172.21.15.86:0/514939716 wait complete. 2024-03-29T11:29:47.826 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:47.833+0000 7f81d4339640 1 librados: shutdown 2024-03-29T11:29:47.842 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:29:54.648 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:29:54.801 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.807+0000 7fbe4f3d3640 10 monclient: get_monmap_and_config 2024-03-29T11:29:54.801 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.807+0000 7fbe4f3d3640 10 monclient: build_initial_monmap 2024-03-29T11:29:54.801 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.807+0000 7fbe4f3d3640 10 monclient: monmap: 2024-03-29T11:29:54.801 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:29:54.801 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:54.801 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:29:54.809632+0000 2024-03-29T11:29:54.801 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:29:54.809632+0000 2024-03-29T11:29:54.801 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:54.801 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:54.802 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:29:54.802 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:29:54.802 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:29:54.802 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:54.802 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.808+0000 7fbe4f3d3640 1 Processor -- start 2024-03-29T11:29:54.802 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.808+0000 7fbe4f3d3640 1 -- start start 2024-03-29T11:29:54.803 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.808+0000 7fbe4f3d3640 10 monclient: init 2024-03-29T11:29:54.803 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.810+0000 7fbe4f3d3640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:54.803 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.810+0000 7fbe4f3d3640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:29:54.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.813+0000 7fbe4f3d3640 10 monclient(hunting): picked mon.noname-c con 0x7fbe40006890 addr v1:172.21.15.196:6789/0 2024-03-29T11:29:54.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.813+0000 7fbe4e3d1640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fbe40006890 0x7fbe400c0a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:42771/0 (socket says 172.21.15.86:42771) 2024-03-29T11:29:54.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.813+0000 7fbe4e3d1640 1 -- 172.21.15.86:0/2660517140 learned_addr learned my addr 172.21.15.86:0/2660517140 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:29:54.806 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.813+0000 7fbe4f3d3640 10 monclient(hunting): picked mon.noname-a con 0x7fbe40006c70 addr v1:172.21.15.86:6789/0 2024-03-29T11:29:54.809 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.813+0000 7fbe4f3d3640 10 monclient(hunting): picked mon.noname-b con 0x7fbe4002b4c0 addr v1:172.21.15.86:6790/0 2024-03-29T11:29:54.809 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.815+0000 7fbe4f3d3640 1 -- 172.21.15.86:0/2660517140 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe4002c830 con 0x7fbe40006c70 2024-03-29T11:29:54.809 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.815+0000 7fbe4f3d3640 1 -- 172.21.15.86:0/2660517140 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe4002afa0 con 0x7fbe40006890 2024-03-29T11:29:54.809 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.815+0000 7fbe4f3d3640 1 -- 172.21.15.86:0/2660517140 --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe4002aba0 con 0x7fbe4002b4c0 2024-03-29T11:29:54.809 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.815+0000 7fbe4f3d3640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:54.809 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.815+0000 7fbe4f3d3640 10 monclient(hunting): authenticate will time out at 1419.134465s 2024-03-29T11:29:54.809 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.816+0000 7fbe4d3cf640 1 -- 172.21.15.86:0/2660517140 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1760976497 0 0) 0x7fbe4002afa0 con 0x7fbe40006890 2024-03-29T11:29:54.809 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.816+0000 7fbe4d3cf640 10 monclient(hunting): _negotiate 2024-03-29T11:29:54.809 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.816+0000 7fbe4d3cf640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:54.809 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.816+0000 7fbe4d3cf640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:54.809 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.816+0000 7fbe4d3cf640 10 monclient(hunting): my global_id is 4504 2024-03-29T11:29:54.809 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.816+0000 7fbe4d3cf640 1 -- 172.21.15.86:0/2660517140 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe34003520 con 0x7fbe40006890 2024-03-29T11:29:54.810 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.816+0000 7fbe4d3cf640 1 -- 172.21.15.86:0/2660517140 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4178396507 0 0) 0x7fbe34003520 con 0x7fbe40006890 2024-03-29T11:29:54.810 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.816+0000 7fbe4d3cf640 1 -- 172.21.15.86:0/2660517140 >> v1:172.21.15.86:6790/0 conn(0x7fbe4002b4c0 legacy=0x7fbe400cf0e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:54.810 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.818+0000 7fbe4d3cf640 1 -- 172.21.15.86:0/2660517140 >> v1:172.21.15.86:6789/0 conn(0x7fbe40006c70 legacy=0x7fbe400c6d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:29:54.811 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.818+0000 7fbe4d3cf640 10 monclient: _finish_hunting 0 2024-03-29T11:29:54.811 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.818+0000 7fbe4d3cf640 1 monclient: found mon.noname-c 2024-03-29T11:29:54.811 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.818+0000 7fbe4d3cf640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:54.813 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.819+0000 7fbe4d3cf640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T11:29:54.813 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.819+0000 7fbe4d3cf640 1 -- 172.21.15.86:0/2660517140 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe4002ad30 con 0x7fbe40006890 2024-03-29T11:29:54.813 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.819+0000 7fbe4d3cf640 10 monclient: _finish_auth 0 2024-03-29T11:29:54.813 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.819+0000 7fbe4d3cf640 10 monclient: _check_auth_tickets 2024-03-29T11:29:54.813 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.819+0000 7fbe4d3cf640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:54.813 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.819+0000 7fbe4d3cf640 1 -- 172.21.15.86:0/2660517140 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fbe4804f730 con 0x7fbe40006890 2024-03-29T11:29:54.813 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.819+0000 7fbe4f3d3640 5 monclient: authenticate success, global_id 4504 2024-03-29T11:29:54.813 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.819+0000 7fbe4f3d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:54.813 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.819+0000 7fbe4d3cf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:54.813 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.819+0000 7fbe4d3cf640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:29:54.813 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.820+0000 7fbe4d3cf640 10 monclient: dump: 2024-03-29T11:29:54.813 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:29:54.813 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:54.813 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:54.814 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:54.814 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:54.814 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:54.814 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:29:54.814 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:29:54.814 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:29:54.814 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:54.814 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.820+0000 7fbe4f3d3640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:29:54.817 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.823+0000 7fbe4d3cf640 1 -- 172.21.15.86:0/2660517140 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe480008d0 con 0x7fbe40006890 2024-03-29T11:29:54.817 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.824+0000 7fbe4d3cf640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:54.817 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.824+0000 7fbe4d3cf640 1 -- 172.21.15.86:0/2660517140 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fbe480758b0 con 0x7fbe40006890 2024-03-29T11:29:54.817 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.824+0000 7fbe4d3cf640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:54.817 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.824+0000 7fbe4d3cf640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:54.817 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.824+0000 7fbe4d3cf640 10 monclient: dump: 2024-03-29T11:29:54.817 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:29:54.817 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:54.817 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:54.817 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:54.817 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:54.817 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:54.817 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:29:54.817 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:29:54.818 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:29:54.818 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:54.818 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.824+0000 7fbe4f3d3640 10 monclient: get_monmap_and_config success 2024-03-29T11:29:54.818 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.824+0000 7fbe4f3d3640 4 set_mon_vals no callback set 2024-03-29T11:29:54.818 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.825+0000 7fbe4f3d3640 10 monclient: shutdown 2024-03-29T11:29:54.818 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.825+0000 7fbe4f3d3640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:54.818 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.825+0000 7fbe4f3d3640 1 -- 172.21.15.86:0/2660517140 >> v1:172.21.15.196:6789/0 conn(0x7fbe40006890 legacy=0x7fbe400c0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:54.819 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.826+0000 7fbe4f3d3640 1 -- 172.21.15.86:0/2660517140 shutdown_connections 2024-03-29T11:29:54.819 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.827+0000 7fbe4f3d3640 1 -- 172.21.15.86:0/2660517140 >> 172.21.15.86:0/2660517140 conn(0x7fbe400058f0 msgr2=0x7fbe40005d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:54.823 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.830+0000 7fbe4f3d3640 1 -- 172.21.15.86:0/2660517140 shutdown_connections 2024-03-29T11:29:54.823 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.830+0000 7fbe4f3d3640 1 -- 172.21.15.86:0/2660517140 wait complete. 2024-03-29T11:29:54.824 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.831+0000 7fbe4f3d3640 10 monclient: build_initial_monmap 2024-03-29T11:29:54.824 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.831+0000 7fbe4f3d3640 10 monclient: monmap: 2024-03-29T11:29:54.824 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:29:54.824 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:29:54.824 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:29:54.825 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:29:54.825 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:29:54.825 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:54.825 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:29:54.825 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:29:54.825 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:29:54.825 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:54.826 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.832+0000 7fbe4f3d3640 1 librados: starting msgr at 2024-03-29T11:29:54.826 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.832+0000 7fbe4f3d3640 1 librados: starting objecter 2024-03-29T11:29:54.826 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.832+0000 7fbe4f3d3640 1 Processor -- start 2024-03-29T11:29:54.826 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.833+0000 7fbe4f3d3640 1 -- start start 2024-03-29T11:29:54.826 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.833+0000 7fbe4f3d3640 1 librados: setting wanted keys 2024-03-29T11:29:54.827 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.833+0000 7fbe4f3d3640 1 librados: calling monclient init 2024-03-29T11:29:54.827 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.833+0000 7fbe4f3d3640 10 monclient: init 2024-03-29T11:29:54.827 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.833+0000 7fbe4f3d3640 10 monclient: _reopen_session rank -1 2024-03-29T11:29:54.828 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.833+0000 7fbe4f3d3640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:29:54.828 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.833+0000 7fbe4f3d3640 10 monclient(hunting): picked mon.noname-b con 0x7fbe40006890 addr v1:172.21.15.196:6789/0 2024-03-29T11:29:54.828 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.833+0000 7fbe4f3d3640 10 monclient(hunting): picked mon.noname-c con 0x7fbe4002b4c0 addr v1:172.21.15.86:6790/0 2024-03-29T11:29:54.828 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.833+0000 7fbe4f3d3640 10 monclient(hunting): picked mon.noname-a con 0x7fbe400c3360 addr v1:172.21.15.86:6789/0 2024-03-29T11:29:54.829 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.833+0000 7fbe4f3d3640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe4002fce0 con 0x7fbe400c3360 2024-03-29T11:29:54.829 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.833+0000 7fbe4f3d3640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe4002ff30 con 0x7fbe40006890 2024-03-29T11:29:54.829 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.833+0000 7fbe4f3d3640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbe40030210 con 0x7fbe4002b4c0 2024-03-29T11:29:54.829 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.833+0000 7fbe4f3d3640 10 monclient(hunting): _renew_subs 2024-03-29T11:29:54.829 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.833+0000 7fbe4f3d3640 10 monclient(hunting): authenticate will time out at 1419.152952s 2024-03-29T11:29:54.832 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.837+0000 7fbe4ebd2640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7fbe400c3360 0x7fbe40165570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:54009/0 (socket says 172.21.15.86:54009) 2024-03-29T11:29:54.832 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.838+0000 7fbe4ebd2640 1 -- 172.21.15.86:0/470417254 learned_addr learned my addr 172.21.15.86:0/470417254 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:29:54.836 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.840+0000 7fbe3effd640 10 client.?.objecter ms_handle_connect 0x7fbe400c3360 2024-03-29T11:29:54.836 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.840+0000 7fbe3effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:54.836 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.840+0000 7fbe3effd640 10 client.?.objecter ms_handle_connect 0x7fbe40006890 2024-03-29T11:29:54.836 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.840+0000 7fbe3effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:54.836 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.841+0000 7fbe3effd640 1 -- 172.21.15.86:0/470417254 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1251410309 0 0) 0x7fbe4002ff30 con 0x7fbe40006890 2024-03-29T11:29:54.836 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.841+0000 7fbe3effd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:54.836 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.841+0000 7fbe3effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:54.836 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.841+0000 7fbe3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:54.836 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.841+0000 7fbe3effd640 10 monclient(hunting): my global_id is 4507 2024-03-29T11:29:54.837 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.841+0000 7fbe3effd640 1 -- 172.21.15.86:0/470417254 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe2c004eb0 con 0x7fbe40006890 2024-03-29T11:29:54.837 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.841+0000 7fbe3effd640 10 client.?.objecter ms_handle_connect 0x7fbe4002b4c0 2024-03-29T11:29:54.837 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.841+0000 7fbe3effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:29:54.837 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.841+0000 7fbe3effd640 1 -- 172.21.15.86:0/470417254 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 568643251 0 0) 0x7fbe4002fce0 con 0x7fbe400c3360 2024-03-29T11:29:54.837 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.841+0000 7fbe3effd640 10 monclient(hunting): _negotiate 2024-03-29T11:29:54.837 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.841+0000 7fbe3effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:29:54.837 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.841+0000 7fbe3effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:29:54.837 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.841+0000 7fbe3effd640 10 monclient(hunting): my global_id is 4536 2024-03-29T11:29:54.837 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.841+0000 7fbe3effd640 1 -- 172.21.15.86:0/470417254 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe4002ff30 con 0x7fbe400c3360 2024-03-29T11:29:54.837 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.841+0000 7fbe3effd640 1 -- 172.21.15.86:0/470417254 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2811006029 0 0) 0x7fbe2c004eb0 con 0x7fbe40006890 2024-03-29T11:29:54.837 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.841+0000 7fbe3effd640 1 -- 172.21.15.86:0/470417254 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fbe4002fce0 con 0x7fbe40006890 2024-03-29T11:29:54.837 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.842+0000 7fbe3effd640 1 -- 172.21.15.86:0/470417254 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fbe4804f400 con 0x7fbe40006890 2024-03-29T11:29:54.837 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.842+0000 7fbe3effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:54.837 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.842+0000 7fbe3effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:29:54.837 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.842+0000 7fbe3effd640 10 monclient(hunting): dump: 2024-03-29T11:29:54.837 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:29:54.838 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:54.838 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:54.838 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:54.838 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:54.838 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:54.838 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:29:54.838 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:29:54.838 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:29:54.838 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:54.838 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.842+0000 7fbe3effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:29:54.838 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.842+0000 7fbe3effd640 1 -- 172.21.15.86:0/470417254 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2299044087 0 0) 0x7fbe4002fce0 con 0x7fbe40006890 2024-03-29T11:29:54.838 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.842+0000 7fbe3effd640 1 -- 172.21.15.86:0/470417254 >> v1:172.21.15.86:6790/0 conn(0x7fbe4002b4c0 legacy=0x7fbe400b13a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:54.838 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.842+0000 7fbe3effd640 1 -- 172.21.15.86:0/470417254 >> v1:172.21.15.86:6789/0 conn(0x7fbe400c3360 legacy=0x7fbe40165570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:54.839 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.842+0000 7fbe3effd640 10 monclient: _finish_hunting 0 2024-03-29T11:29:54.839 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.842+0000 7fbe3effd640 1 monclient: found mon.b 2024-03-29T11:29:54.839 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.842+0000 7fbe3effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:29:54.839 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.842+0000 7fbe3effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:29:54.839 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.842+0000 7fbe3effd640 1 -- 172.21.15.86:0/470417254 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe400d71f0 con 0x7fbe40006890 2024-03-29T11:29:54.839 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.842+0000 7fbe3effd640 10 monclient: _finish_auth 0 2024-03-29T11:29:54.839 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.842+0000 7fbe3effd640 10 monclient: _check_auth_tickets 2024-03-29T11:29:54.839 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.842+0000 7fbe3effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:29:54.839 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.842+0000 7fbe4f3d3640 5 monclient: authenticate success, global_id 4507 2024-03-29T11:29:54.839 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.842+0000 7fbe4f3d3640 10 monclient: _renew_subs 2024-03-29T11:29:54.839 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.842+0000 7fbe4f3d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:29:54.839 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.842+0000 7fbe4f3d3640 1 -- 172.21.15.86:0/470417254 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe400304f0 con 0x7fbe40006890 2024-03-29T11:29:54.839 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.842+0000 7fbe4f3d3640 10 client.4507.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:29:54.839 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.842+0000 7fbe4f3d3640 10 monclient: _renew_subs 2024-03-29T11:29:54.840 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.843+0000 7fbe4f3d3640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:29:54.840 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.843+0000 7fbe4f3d3640 1 -- 172.21.15.86:0/470417254 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe4002ccf0 con 0x7fbe40006890 2024-03-29T11:29:54.840 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.843+0000 7fbe4f3d3640 1 librados: init done 2024-03-29T11:29:54.840 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.844+0000 7fbe3effd640 1 -- 172.21.15.86:0/470417254 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fbe4804f730 con 0x7fbe40006890 2024-03-29T11:29:54.840 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.844+0000 7fbe3effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:29:54.840 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.844+0000 7fbe3effd640 1 -- 172.21.15.86:0/470417254 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fbe48067d90 con 0x7fbe40006890 2024-03-29T11:29:54.840 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.844+0000 7fbe3effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:29:54.840 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.844+0000 7fbe3effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:29:54.840 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.844+0000 7fbe3effd640 10 monclient: dump: 2024-03-29T11:29:54.840 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:29:54.841 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:29:54.841 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:54.841 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:29:54.841 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:29:54.841 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:29:54.841 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:29:54.841 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:29:54.841 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:29:54.841 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:29:54.841 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.845+0000 7fbe3ffff640 4 set_mon_vals no callback set 2024-03-29T11:29:54.841 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.845+0000 7fbe3effd640 1 -- 172.21.15.86:0/470417254 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7fbe48067fb0 con 0x7fbe40006890 2024-03-29T11:29:54.841 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.846+0000 7fbe3effd640 1 -- 172.21.15.86:0/470417254 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7fbe480ff3f0 con 0x7fbe40006890 2024-03-29T11:29:54.841 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.846+0000 7fbe3effd640 10 client.4507.objecter ms_dispatch 0x7fbe48000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:29:54.841 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.846+0000 7fbe3effd640 3 client.4507.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:29:54.841 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.846+0000 7fbe3effd640 3 client.4507.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:29:54.842 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.846+0000 7fbe3effd640 20 client.4507.objecter dump_active .. 0 homeless 2024-03-29T11:29:54.842 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.848+0000 7fbe4dbd0640 10 monclient: get_auth_request con 0x7fbe2c07ec30 auth_method 0 2024-03-29T11:29:54.843 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.848+0000 7fbe3cff9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:54.843 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.848+0000 7fbe3cff9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:54.843 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.849+0000 7fbe3cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:29:54.843 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.849+0000 7fbe3effd640 10 client.4507.objecter ms_handle_connect 0x7fbe2c07ec30 2024-03-29T11:29:54.843 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.849+0000 7fbe3cff9640 1 -- 172.21.15.86:0/470417254 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe30005080 con 0x7fbe40006890 2024-03-29T11:29:54.848 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.855+0000 7fbe3effd640 1 -- 172.21.15.86:0/470417254 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fbe480cacd0 con 0x7fbe40006890 2024-03-29T11:29:54.848 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.856+0000 7fbe3effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:29:54.849 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:54.856+0000 7fbe3effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:29:55.227 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:55.234+0000 7fbe3cff9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:55.228 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:55.234+0000 7fbe3cff9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:55.228 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:55.234+0000 7fbe3cff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:29:55.228 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:55.234+0000 7fbe3cff9640 1 -- 172.21.15.86:0/470417254 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbe30005330 con 0x7fbe40006890 2024-03-29T11:29:55.228 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:55.235+0000 7fbe3effd640 1 -- 172.21.15.86:0/470417254 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7fbe480ccc00 con 0x7fbe40006890 2024-03-29T11:29:55.229 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:55.235+0000 7fbe3effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:29:55.229 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:55.235+0000 7fbe3effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:29:55.229 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:29:55.229 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:29:55.232 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:55.239+0000 7fbe3cff9640 10 librados: watch_flush enter 2024-03-29T11:29:55.266 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:55.239+0000 7fbe3cff9640 10 librados: watch_flush exit 2024-03-29T11:29:55.266 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:55.239+0000 7fbe3cff9640 20 client.4507.objecter shutdown clearing up homeless session... 2024-03-29T11:29:55.267 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:55.239+0000 7fbe3cff9640 10 client.4507.objecter successfully canceled tick 2024-03-29T11:29:55.268 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:55.240+0000 7fbe3cff9640 1 -- 172.21.15.86:0/470417254 >> v1:172.21.15.196:6816/588110318 conn(0x7fbe2c07ec30 legacy=0x7fbe2c0810f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:55.268 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:55.240+0000 7fbe3cff9640 10 monclient: shutdown 2024-03-29T11:29:55.268 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:55.240+0000 7fbe3cff9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:29:55.268 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:55.240+0000 7fbe3cff9640 1 -- 172.21.15.86:0/470417254 >> v1:172.21.15.196:6789/0 conn(0x7fbe40006890 legacy=0x7fbe4002d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:29:55.268 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:55.240+0000 7fbe3cff9640 1 -- 172.21.15.86:0/470417254 shutdown_connections 2024-03-29T11:29:55.268 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:55.240+0000 7fbe3cff9640 1 -- 172.21.15.86:0/470417254 >> 172.21.15.86:0/470417254 conn(0x7fbe400058f0 msgr2=0x7fbe40038130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:29:55.268 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:55.240+0000 7fbe3cff9640 1 -- 172.21.15.86:0/470417254 shutdown_connections 2024-03-29T11:29:55.268 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:55.241+0000 7fbe3cff9640 1 -- 172.21.15.86:0/470417254 wait complete. 2024-03-29T11:29:55.269 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:29:55.241+0000 7fbe3cff9640 1 librados: shutdown 2024-03-29T11:29:55.269 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:02.080 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:02.233 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.241+0000 7f47cab84640 10 monclient: get_monmap_and_config 2024-03-29T11:30:02.233 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.241+0000 7f47cab84640 10 monclient: build_initial_monmap 2024-03-29T11:30:02.233 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.241+0000 7f47cab84640 10 monclient: monmap: 2024-03-29T11:30:02.233 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:30:02.233 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:02.233 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:30:02.242239+0000 2024-03-29T11:30:02.234 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:30:02.242239+0000 2024-03-29T11:30:02.234 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:02.234 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:02.234 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:30:02.234 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:30:02.234 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:30:02.234 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:02.236 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.242+0000 7f47cab84640 1 Processor -- start 2024-03-29T11:30:02.236 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.242+0000 7f47cab84640 1 -- start start 2024-03-29T11:30:02.236 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.242+0000 7f47cab84640 10 monclient: init 2024-03-29T11:30:02.237 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.243+0000 7f47cab84640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:02.237 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.243+0000 7f47cab84640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:30:02.237 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.243+0000 7f47cab84640 10 monclient(hunting): picked mon.noname-a con 0x7f47c40b0770 addr v1:172.21.15.86:6789/0 2024-03-29T11:30:02.244 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.249+0000 7f47c8920640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f47c40b0770 0x7f47c4149ff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:37325/0 (socket says 172.21.15.86:37325) 2024-03-29T11:30:02.244 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.249+0000 7f47cab84640 10 monclient(hunting): picked mon.noname-c con 0x7f47c414a700 addr v1:172.21.15.196:6789/0 2024-03-29T11:30:02.244 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.249+0000 7f47c8920640 1 -- 172.21.15.86:0/4098909246 learned_addr learned my addr 172.21.15.86:0/4098909246 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:30:02.244 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.251+0000 7f47cab84640 10 monclient(hunting): picked mon.noname-b con 0x7f47c40ac5a0 addr v1:172.21.15.86:6790/0 2024-03-29T11:30:02.244 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.251+0000 7f47cab84640 1 -- 172.21.15.86:0/4098909246 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47c40aba70 con 0x7f47c40b0770 2024-03-29T11:30:02.244 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.251+0000 7f47cab84640 1 -- 172.21.15.86:0/4098909246 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47c40aecb0 con 0x7f47c414a700 2024-03-29T11:30:02.244 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.251+0000 7f47cab84640 1 -- 172.21.15.86:0/4098909246 --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47c40af150 con 0x7f47c40ac5a0 2024-03-29T11:30:02.244 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.251+0000 7f47cab84640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:02.244 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.251+0000 7f47cab84640 10 monclient(hunting): authenticate will time out at 1426.569607s 2024-03-29T11:30:02.246 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.252+0000 7f47c37fe640 1 -- 172.21.15.86:0/4098909246 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2385527222 0 0) 0x7f47c40aecb0 con 0x7f47c414a700 2024-03-29T11:30:02.246 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.252+0000 7f47c37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:02.246 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.252+0000 7f47c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:02.246 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.252+0000 7f47c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:02.246 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.252+0000 7f47c37fe640 10 monclient(hunting): my global_id is 4510 2024-03-29T11:30:02.246 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.252+0000 7f47c37fe640 1 -- 172.21.15.86:0/4098909246 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47b4003520 con 0x7f47c414a700 2024-03-29T11:30:02.246 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.252+0000 7f47c37fe640 1 -- 172.21.15.86:0/4098909246 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3847363249 0 0) 0x7f47c40af150 con 0x7f47c40ac5a0 2024-03-29T11:30:02.246 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.252+0000 7f47c37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:02.247 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.252+0000 7f47c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:02.247 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.252+0000 7f47c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:02.247 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.252+0000 7f47c37fe640 10 monclient(hunting): my global_id is 4628 2024-03-29T11:30:02.247 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.252+0000 7f47c37fe640 1 -- 172.21.15.86:0/4098909246 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47c40aecb0 con 0x7f47c40ac5a0 2024-03-29T11:30:02.247 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.252+0000 7f47c37fe640 1 -- 172.21.15.86:0/4098909246 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2899678836 0 0) 0x7f47c40aba70 con 0x7f47c40b0770 2024-03-29T11:30:02.247 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.252+0000 7f47c37fe640 10 monclient(hunting): _negotiate 2024-03-29T11:30:02.247 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.252+0000 7f47c37fe640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:02.247 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.252+0000 7f47c37fe640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:02.247 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.252+0000 7f47c37fe640 10 monclient(hunting): my global_id is 4539 2024-03-29T11:30:02.247 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.252+0000 7f47c37fe640 1 -- 172.21.15.86:0/4098909246 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47c40af150 con 0x7f47c40b0770 2024-03-29T11:30:02.247 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.253+0000 7f47c37fe640 1 -- 172.21.15.86:0/4098909246 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 272692045 0 0) 0x7f47b4003520 con 0x7f47c414a700 2024-03-29T11:30:02.247 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.253+0000 7f47c37fe640 1 -- 172.21.15.86:0/4098909246 >> v1:172.21.15.86:6790/0 conn(0x7f47c40ac5a0 legacy=0x7f47c4151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:02.248 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.253+0000 7f47c37fe640 1 -- 172.21.15.86:0/4098909246 >> v1:172.21.15.86:6789/0 conn(0x7f47c40b0770 legacy=0x7f47c4149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:02.248 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.253+0000 7f47c37fe640 10 monclient: _finish_hunting 0 2024-03-29T11:30:02.248 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.253+0000 7f47c37fe640 1 monclient: found mon.noname-c 2024-03-29T11:30:02.248 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.253+0000 7f47c37fe640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:02.248 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.253+0000 7f47c37fe640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T11:30:02.248 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.253+0000 7f47c37fe640 1 -- 172.21.15.86:0/4098909246 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c40ac9d0 con 0x7f47c414a700 2024-03-29T11:30:02.248 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.253+0000 7f47c37fe640 10 monclient: _finish_auth 0 2024-03-29T11:30:02.248 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.253+0000 7f47c37fe640 10 monclient: _check_auth_tickets 2024-03-29T11:30:02.248 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.253+0000 7f47c37fe640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:02.248 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.253+0000 7f47cab84640 5 monclient: authenticate success, global_id 4510 2024-03-29T11:30:02.248 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.253+0000 7f47cab84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:02.249 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.253+0000 7f47c37fe640 1 -- 172.21.15.86:0/4098909246 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f47b00016b0 con 0x7f47c414a700 2024-03-29T11:30:02.249 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.253+0000 7f47c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:02.249 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.253+0000 7f47c37fe640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:02.249 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.253+0000 7f47c37fe640 10 monclient: dump: 2024-03-29T11:30:02.249 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:30:02.249 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:02.249 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:02.249 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:02.249 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:02.249 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:02.249 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:30:02.249 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:30:02.249 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:30:02.249 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:02.249 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.253+0000 7f47cab84640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:02.250 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.256+0000 7f47c37fe640 1 -- 172.21.15.86:0/4098909246 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47b0001900 con 0x7f47c414a700 2024-03-29T11:30:02.250 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.256+0000 7f47c37fe640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:02.250 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.256+0000 7f47c37fe640 1 -- 172.21.15.86:0/4098909246 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f47b0001c90 con 0x7f47c414a700 2024-03-29T11:30:02.250 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.256+0000 7f47c37fe640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:02.251 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.256+0000 7f47c37fe640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:02.251 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.256+0000 7f47c37fe640 10 monclient: dump: 2024-03-29T11:30:02.251 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:30:02.251 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:02.251 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:02.251 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:02.251 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:02.251 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:02.251 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:30:02.251 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:30:02.251 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:30:02.251 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:02.252 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.256+0000 7f47cab84640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:02.252 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.256+0000 7f47cab84640 4 set_mon_vals no callback set 2024-03-29T11:30:02.252 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.257+0000 7f47cab84640 10 monclient: shutdown 2024-03-29T11:30:02.252 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.257+0000 7f47cab84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:02.252 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.257+0000 7f47cab84640 1 -- 172.21.15.86:0/4098909246 >> v1:172.21.15.196:6789/0 conn(0x7f47c414a700 legacy=0x7f47c40abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:02.252 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.258+0000 7f47cab84640 1 -- 172.21.15.86:0/4098909246 shutdown_connections 2024-03-29T11:30:02.252 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.258+0000 7f47cab84640 1 -- 172.21.15.86:0/4098909246 >> 172.21.15.86:0/4098909246 conn(0x7f47c406b660 msgr2=0x7f47c406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:02.253 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.258+0000 7f47cab84640 1 -- 172.21.15.86:0/4098909246 shutdown_connections 2024-03-29T11:30:02.253 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.258+0000 7f47cab84640 1 -- 172.21.15.86:0/4098909246 wait complete. 2024-03-29T11:30:02.253 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.259+0000 7f47cab84640 10 monclient: build_initial_monmap 2024-03-29T11:30:02.253 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.259+0000 7f47cab84640 10 monclient: monmap: 2024-03-29T11:30:02.253 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:30:02.253 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:02.253 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:30:02.253 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:30:02.254 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:02.254 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:02.254 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:30:02.254 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:30:02.254 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:30:02.254 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:02.254 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.260+0000 7f47cab84640 1 librados: starting msgr at 2024-03-29T11:30:02.254 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.260+0000 7f47cab84640 1 librados: starting objecter 2024-03-29T11:30:02.254 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.260+0000 7f47cab84640 1 Processor -- start 2024-03-29T11:30:02.254 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.260+0000 7f47cab84640 1 -- start start 2024-03-29T11:30:02.254 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.260+0000 7f47cab84640 1 librados: setting wanted keys 2024-03-29T11:30:02.254 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.260+0000 7f47cab84640 1 librados: calling monclient init 2024-03-29T11:30:02.254 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.260+0000 7f47cab84640 10 monclient: init 2024-03-29T11:30:02.254 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.261+0000 7f47cab84640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:02.254 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.261+0000 7f47cab84640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:30:02.255 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.261+0000 7f47cab84640 10 monclient(hunting): picked mon.noname-b con 0x7f47c40ac5a0 addr v1:172.21.15.196:6789/0 2024-03-29T11:30:02.257 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.262+0000 7f47cab84640 10 monclient(hunting): picked mon.noname-c con 0x7f47c40b0770 addr v1:172.21.15.86:6790/0 2024-03-29T11:30:02.257 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.262+0000 7f47cab84640 10 monclient(hunting): picked mon.noname-a con 0x7f47c414a700 addr v1:172.21.15.86:6789/0 2024-03-29T11:30:02.257 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.262+0000 7f47cab84640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47c41208b0 con 0x7f47c414a700 2024-03-29T11:30:02.257 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.262+0000 7f47cab84640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47c411f7c0 con 0x7f47c40ac5a0 2024-03-29T11:30:02.257 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.262+0000 7f47cab84640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47c411faa0 con 0x7f47c40b0770 2024-03-29T11:30:02.258 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.262+0000 7f47cab84640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:02.258 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.262+0000 7f47cab84640 10 monclient(hunting): authenticate will time out at 1426.580855s 2024-03-29T11:30:02.258 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.264+0000 7f47c8920640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f47c40ac5a0 0x7f47c41201a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:35477/0 (socket says 172.21.15.86:35477) 2024-03-29T11:30:02.258 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.264+0000 7f47c8920640 1 -- 172.21.15.86:0/2015196038 learned_addr learned my addr 172.21.15.86:0/2015196038 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:30:02.259 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.267+0000 7f47c17fa640 10 client.?.objecter ms_handle_connect 0x7f47c40ac5a0 2024-03-29T11:30:02.259 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.267+0000 7f47c17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:02.260 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.267+0000 7f47c17fa640 10 client.?.objecter ms_handle_connect 0x7f47c40b0770 2024-03-29T11:30:02.260 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.268+0000 7f47c17fa640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:02.260 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.268+0000 7f47c17fa640 1 -- 172.21.15.86:0/2015196038 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1675855855 0 0) 0x7f47c411f7c0 con 0x7f47c40ac5a0 2024-03-29T11:30:02.260 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.268+0000 7f47c17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:30:02.260 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.268+0000 7f47c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:02.260 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.268+0000 7f47c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:02.260 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.268+0000 7f47c17fa640 10 monclient(hunting): my global_id is 4513 2024-03-29T11:30:02.260 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.268+0000 7f47c17fa640 1 -- 172.21.15.86:0/2015196038 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47a8004eb0 con 0x7f47c40ac5a0 2024-03-29T11:30:02.261 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.268+0000 7f47c17fa640 1 -- 172.21.15.86:0/2015196038 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 672447022 0 0) 0x7f47c411faa0 con 0x7f47c40b0770 2024-03-29T11:30:02.261 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.268+0000 7f47c17fa640 10 monclient(hunting): _negotiate 2024-03-29T11:30:02.261 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.268+0000 7f47c17fa640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:02.261 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.268+0000 7f47c17fa640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:02.261 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.268+0000 7f47c17fa640 10 monclient(hunting): my global_id is 4631 2024-03-29T11:30:02.261 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.268+0000 7f47c17fa640 1 -- 172.21.15.86:0/2015196038 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47c411f7c0 con 0x7f47c40b0770 2024-03-29T11:30:02.261 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.268+0000 7f47c17fa640 1 -- 172.21.15.86:0/2015196038 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 149277782 0 0) 0x7f47c411f7c0 con 0x7f47c40b0770 2024-03-29T11:30:02.261 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.268+0000 7f47c17fa640 1 -- 172.21.15.86:0/2015196038 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f47c411faa0 con 0x7f47c40b0770 2024-03-29T11:30:02.262 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.268+0000 7f47c17fa640 1 -- 172.21.15.86:0/2015196038 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f47b0001c10 con 0x7f47c40b0770 2024-03-29T11:30:02.262 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.268+0000 7f47c17fa640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:02.262 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.268+0000 7f47c17fa640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:02.262 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.268+0000 7f47c17fa640 10 monclient(hunting): dump: 2024-03-29T11:30:02.262 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:30:02.262 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:02.262 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:02.262 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:02.262 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:02.262 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:02.262 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:30:02.262 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:30:02.262 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:30:02.262 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:02.263 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.269+0000 7f47c17fa640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:02.263 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.269+0000 7f47c17fa640 1 -- 172.21.15.86:0/2015196038 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2071085029 0 0) 0x7f47c411faa0 con 0x7f47c40b0770 2024-03-29T11:30:02.263 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.269+0000 7f47c17fa640 1 -- 172.21.15.86:0/2015196038 >> v1:172.21.15.196:6789/0 conn(0x7f47c40ac5a0 legacy=0x7f47c41201a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:02.263 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.269+0000 7f47c17fa640 1 -- 172.21.15.86:0/2015196038 >> v1:172.21.15.86:6789/0 conn(0x7f47c414a700 legacy=0x7f47c41ffd40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:30:02.264 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.271+0000 7f47c17fa640 10 monclient: _finish_hunting 0 2024-03-29T11:30:02.264 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.271+0000 7f47c17fa640 1 monclient: found mon.c 2024-03-29T11:30:02.264 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.271+0000 7f47c17fa640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:02.264 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.271+0000 7f47c17fa640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:30:02.264 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.271+0000 7f47c17fa640 1 -- 172.21.15.86:0/2015196038 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c411d900 con 0x7f47c40b0770 2024-03-29T11:30:02.264 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.271+0000 7f47c17fa640 10 monclient: _finish_auth 0 2024-03-29T11:30:02.264 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.271+0000 7f47c17fa640 10 monclient: _check_auth_tickets 2024-03-29T11:30:02.264 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.271+0000 7f47c17fa640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:02.264 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.271+0000 7f47cab84640 5 monclient: authenticate success, global_id 4631 2024-03-29T11:30:02.264 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.271+0000 7f47cab84640 10 monclient: _renew_subs 2024-03-29T11:30:02.264 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.271+0000 7f47cab84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:30:02.264 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.271+0000 7f47cab84640 1 -- 172.21.15.86:0/2015196038 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47c411e960 con 0x7f47c40b0770 2024-03-29T11:30:02.264 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.271+0000 7f47cab84640 10 client.4631.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:02.265 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.271+0000 7f47cab84640 10 monclient: _renew_subs 2024-03-29T11:30:02.265 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.271+0000 7f47cab84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:30:02.265 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.271+0000 7f47cab84640 1 -- 172.21.15.86:0/2015196038 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f47c411c2d0 con 0x7f47c40b0770 2024-03-29T11:30:02.265 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.271+0000 7f47cab84640 1 librados: init done 2024-03-29T11:30:02.266 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.271+0000 7f47c17fa640 1 -- 172.21.15.86:0/2015196038 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f47b0001250 con 0x7f47c40b0770 2024-03-29T11:30:02.266 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.271+0000 7f47c17fa640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:02.266 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.272+0000 7f47c2ffd640 4 set_mon_vals no callback set 2024-03-29T11:30:02.266 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.272+0000 7f47c17fa640 1 -- 172.21.15.86:0/2015196038 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f47b0003a40 con 0x7f47c40b0770 2024-03-29T11:30:02.266 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.272+0000 7f47c17fa640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:02.266 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.272+0000 7f47c17fa640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:02.266 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.272+0000 7f47c17fa640 10 monclient: dump: 2024-03-29T11:30:02.266 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:30:02.266 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:02.266 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:02.266 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:02.266 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:02.267 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:02.267 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:30:02.267 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:30:02.267 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:30:02.267 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:02.267 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.272+0000 7f47cab84640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:02.267 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.272+0000 7f47cab84640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:02.267 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.272+0000 7f47cab84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:30:02.267 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.272+0000 7f47cab84640 1 -- 172.21.15.86:0/2015196038 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a4005080 con 0x7f47c40b0770 2024-03-29T11:30:02.267 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.274+0000 7f47c17fa640 1 -- 172.21.15.86:0/2015196038 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f47b0019a10 con 0x7f47c40b0770 2024-03-29T11:30:02.268 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.274+0000 7f47c17fa640 1 -- 172.21.15.86:0/2015196038 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f47b00838e0 con 0x7f47c40b0770 2024-03-29T11:30:02.268 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.275+0000 7f47c17fa640 10 client.4631.objecter ms_dispatch 0x7f47c4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:30:02.268 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.275+0000 7f47c17fa640 3 client.4631.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:30:02.268 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.275+0000 7f47c17fa640 3 client.4631.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:30:02.268 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.275+0000 7f47c17fa640 20 client.4631.objecter dump_active .. 0 homeless 2024-03-29T11:30:02.269 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.277+0000 7f47c8920640 10 monclient: get_auth_request con 0x7f47a806dc90 auth_method 0 2024-03-29T11:30:02.271 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.278+0000 7f47c17fa640 10 client.4631.objecter ms_handle_connect 0x7f47a806dc90 2024-03-29T11:30:02.277 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.285+0000 7f47c17fa640 1 -- 172.21.15.86:0/2015196038 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f47b004f1c0 con 0x7f47c40b0770 2024-03-29T11:30:02.277 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.285+0000 7f47c17fa640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:02.277 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.285+0000 7f47c17fa640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:02.607 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.614+0000 7f47cab84640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:02.608 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.614+0000 7f47cab84640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:02.608 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.614+0000 7f47cab84640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:30:02.608 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.614+0000 7f47cab84640 1 -- 172.21.15.86:0/2015196038 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47a4005330 con 0x7f47c40b0770 2024-03-29T11:30:02.608 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.615+0000 7f47c17fa640 1 -- 172.21.15.86:0/2015196038 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f47b00510f0 con 0x7f47c40b0770 2024-03-29T11:30:02.608 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.615+0000 7f47c17fa640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:02.608 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.615+0000 7f47c17fa640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:02.609 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:30:02.609 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:02.613 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.621+0000 7f47cab84640 10 librados: watch_flush enter 2024-03-29T11:30:02.613 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.621+0000 7f47cab84640 10 librados: watch_flush exit 2024-03-29T11:30:02.613 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.621+0000 7f47cab84640 20 client.4631.objecter shutdown clearing up homeless session... 2024-03-29T11:30:02.613 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.621+0000 7f47cab84640 10 client.4631.objecter successfully canceled tick 2024-03-29T11:30:02.613 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.621+0000 7f47cab84640 1 -- 172.21.15.86:0/2015196038 >> v1:172.21.15.196:6816/588110318 conn(0x7f47a806dc90 legacy=0x7f47a8070150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:02.613 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.621+0000 7f47cab84640 10 monclient: shutdown 2024-03-29T11:30:02.613 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.621+0000 7f47cab84640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:02.613 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.621+0000 7f47cab84640 1 -- 172.21.15.86:0/2015196038 >> v1:172.21.15.86:6790/0 conn(0x7f47c40b0770 legacy=0x7f47c411e250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:02.614 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.621+0000 7f47cab84640 1 -- 172.21.15.86:0/2015196038 shutdown_connections 2024-03-29T11:30:02.614 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.621+0000 7f47cab84640 1 -- 172.21.15.86:0/2015196038 >> 172.21.15.86:0/2015196038 conn(0x7f47c406b660 msgr2=0x7f47c40b3e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:02.614 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.622+0000 7f47cab84640 1 -- 172.21.15.86:0/2015196038 shutdown_connections 2024-03-29T11:30:02.614 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.622+0000 7f47cab84640 1 -- 172.21.15.86:0/2015196038 wait complete. 2024-03-29T11:30:02.614 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:02.622+0000 7f47cab84640 1 librados: shutdown 2024-03-29T11:30:02.638 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:09.446 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:09.592 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.599+0000 7f669db32640 10 monclient: get_monmap_and_config 2024-03-29T11:30:09.592 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.599+0000 7f669db32640 10 monclient: build_initial_monmap 2024-03-29T11:30:09.592 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.599+0000 7f669db32640 10 monclient: monmap: 2024-03-29T11:30:09.592 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:30:09.592 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:09.592 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:30:09.600481+0000 2024-03-29T11:30:09.592 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:30:09.600481+0000 2024-03-29T11:30:09.592 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:09.592 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:09.592 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:30:09.592 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:30:09.592 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:30:09.593 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:09.593 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.600+0000 7f669db32640 1 Processor -- start 2024-03-29T11:30:09.593 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.600+0000 7f669db32640 1 -- start start 2024-03-29T11:30:09.593 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.600+0000 7f669db32640 10 monclient: init 2024-03-29T11:30:09.595 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.601+0000 7f669db32640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:09.595 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.601+0000 7f669db32640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:30:09.595 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.601+0000 7f669db32640 10 monclient(hunting): picked mon.noname-c con 0x7f66981422b0 addr v1:172.21.15.196:6789/0 2024-03-29T11:30:09.595 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.601+0000 7f669db32640 10 monclient(hunting): picked mon.noname-a con 0x7f6698142690 addr v1:172.21.15.86:6789/0 2024-03-29T11:30:09.595 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.601+0000 7f669db32640 10 monclient(hunting): picked mon.noname-b con 0x7f66980b17d0 addr v1:172.21.15.86:6790/0 2024-03-29T11:30:09.595 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.601+0000 7f669db32640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6698142c80 con 0x7f6698142690 2024-03-29T11:30:09.596 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.601+0000 7f669db32640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66980aae40 con 0x7f66981422b0 2024-03-29T11:30:09.596 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.601+0000 7f669db32640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f669813fbd0 con 0x7f66980b17d0 2024-03-29T11:30:09.596 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.601+0000 7f669db32640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:09.596 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.601+0000 7f669db32640 10 monclient(hunting): authenticate will time out at 1433.919624s 2024-03-29T11:30:09.596 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.604+0000 7f6697fff640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f66980b17d0 0x7f66980b2ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:39776/0 (socket says 172.21.15.86:39776) 2024-03-29T11:30:09.596 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.604+0000 7f6697fff640 1 -- 172.21.15.86:0/4279410012 learned_addr learned my addr 172.21.15.86:0/4279410012 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:30:09.597 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.604+0000 7f66967fc640 1 -- 172.21.15.86:0/4279410012 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4110354151 0 0) 0x7f66980aae40 con 0x7f66981422b0 2024-03-29T11:30:09.597 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.604+0000 7f66967fc640 10 monclient(hunting): _negotiate 2024-03-29T11:30:09.597 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.604+0000 7f66967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:09.597 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.604+0000 7f66967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:09.597 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.604+0000 7f66967fc640 10 monclient(hunting): my global_id is 4516 2024-03-29T11:30:09.597 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.604+0000 7f66967fc640 1 -- 172.21.15.86:0/4279410012 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6680003520 con 0x7f66981422b0 2024-03-29T11:30:09.598 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.604+0000 7f66967fc640 1 -- 172.21.15.86:0/4279410012 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3084476486 0 0) 0x7f669813fbd0 con 0x7f66980b17d0 2024-03-29T11:30:09.598 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.604+0000 7f66967fc640 10 monclient(hunting): _negotiate 2024-03-29T11:30:09.598 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.604+0000 7f66967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:09.598 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.604+0000 7f66967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:09.598 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.604+0000 7f66967fc640 10 monclient(hunting): my global_id is 4634 2024-03-29T11:30:09.598 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.604+0000 7f66967fc640 1 -- 172.21.15.86:0/4279410012 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f66980aae40 con 0x7f66980b17d0 2024-03-29T11:30:09.598 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.605+0000 7f66967fc640 1 -- 172.21.15.86:0/4279410012 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2436506819 0 0) 0x7f6680003520 con 0x7f66981422b0 2024-03-29T11:30:09.598 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.605+0000 7f66967fc640 1 -- 172.21.15.86:0/4279410012 >> v1:172.21.15.86:6790/0 conn(0x7f66980b17d0 legacy=0x7f66980b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:09.598 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.605+0000 7f66967fc640 1 -- 172.21.15.86:0/4279410012 >> v1:172.21.15.86:6789/0 conn(0x7f6698142690 legacy=0x7f66980b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:30:09.599 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.605+0000 7f66967fc640 10 monclient: _finish_hunting 0 2024-03-29T11:30:09.599 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.606+0000 7f66967fc640 1 monclient: found mon.noname-c 2024-03-29T11:30:09.599 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.606+0000 7f66967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:09.599 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.606+0000 7f66967fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T11:30:09.599 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.606+0000 7f66967fc640 1 -- 172.21.15.86:0/4279410012 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6698151940 con 0x7f66981422b0 2024-03-29T11:30:09.599 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.606+0000 7f66967fc640 10 monclient: _finish_auth 0 2024-03-29T11:30:09.600 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.606+0000 7f66967fc640 10 monclient: _check_auth_tickets 2024-03-29T11:30:09.600 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.606+0000 7f66967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:09.600 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.606+0000 7f669db32640 5 monclient: authenticate success, global_id 4516 2024-03-29T11:30:09.600 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.606+0000 7f669db32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:09.600 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.606+0000 7f66967fc640 1 -- 172.21.15.86:0/4279410012 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f6690001940 con 0x7f66981422b0 2024-03-29T11:30:09.600 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.606+0000 7f66967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:09.600 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.606+0000 7f66967fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:09.600 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.606+0000 7f66967fc640 10 monclient: dump: 2024-03-29T11:30:09.600 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:30:09.600 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:09.600 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:09.600 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:09.601 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:09.601 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:09.601 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:30:09.601 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:30:09.601 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:30:09.601 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:09.601 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.606+0000 7f669db32640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:09.602 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.609+0000 7f66967fc640 1 -- 172.21.15.86:0/4279410012 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6690001aa0 con 0x7f66981422b0 2024-03-29T11:30:09.602 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.609+0000 7f66967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:09.602 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.609+0000 7f669db32640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:09.602 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.609+0000 7f669db32640 4 set_mon_vals no callback set 2024-03-29T11:30:09.603 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.610+0000 7f669db32640 10 monclient: shutdown 2024-03-29T11:30:09.603 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.610+0000 7f669db32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:09.603 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.610+0000 7f669db32640 1 -- 172.21.15.86:0/4279410012 >> v1:172.21.15.196:6789/0 conn(0x7f66981422b0 legacy=0x7f669813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:09.604 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.612+0000 7f669db32640 1 -- 172.21.15.86:0/4279410012 shutdown_connections 2024-03-29T11:30:09.604 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.612+0000 7f669db32640 1 -- 172.21.15.86:0/4279410012 >> 172.21.15.86:0/4279410012 conn(0x7f6698066a00 msgr2=0x7f6698066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:09.604 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.612+0000 7f669db32640 1 -- 172.21.15.86:0/4279410012 shutdown_connections 2024-03-29T11:30:09.605 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.612+0000 7f669db32640 1 -- 172.21.15.86:0/4279410012 wait complete. 2024-03-29T11:30:09.606 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.613+0000 7f669db32640 10 monclient: build_initial_monmap 2024-03-29T11:30:09.606 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.613+0000 7f669db32640 10 monclient: monmap: 2024-03-29T11:30:09.606 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:30:09.606 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:09.606 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:30:09.606 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:30:09.606 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:09.606 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:09.606 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:30:09.607 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:30:09.607 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:30:09.607 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:09.607 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.613+0000 7f669db32640 1 librados: starting msgr at 2024-03-29T11:30:09.607 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.613+0000 7f669db32640 1 librados: starting objecter 2024-03-29T11:30:09.607 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.613+0000 7f669db32640 1 Processor -- start 2024-03-29T11:30:09.609 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.614+0000 7f669db32640 1 -- start start 2024-03-29T11:30:09.609 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.614+0000 7f669db32640 1 librados: setting wanted keys 2024-03-29T11:30:09.609 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.614+0000 7f669db32640 1 librados: calling monclient init 2024-03-29T11:30:09.609 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.614+0000 7f669db32640 10 monclient: init 2024-03-29T11:30:09.609 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.615+0000 7f669db32640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:09.609 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.615+0000 7f669db32640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:30:09.610 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.615+0000 7f669db32640 10 monclient(hunting): picked mon.noname-a con 0x7f66980b17d0 addr v1:172.21.15.86:6789/0 2024-03-29T11:30:09.610 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.615+0000 7f669db32640 10 monclient(hunting): picked mon.noname-c con 0x7f66981422b0 addr v1:172.21.15.86:6790/0 2024-03-29T11:30:09.610 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.615+0000 7f669db32640 10 monclient(hunting): picked mon.noname-b con 0x7f6698142690 addr v1:172.21.15.196:6789/0 2024-03-29T11:30:09.610 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.615+0000 7f669db32640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66981322f0 con 0x7f66980b17d0 2024-03-29T11:30:09.610 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.615+0000 7f669db32640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f669812f680 con 0x7f6698142690 2024-03-29T11:30:09.610 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.615+0000 7f669db32640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f669806f160 con 0x7f66981422b0 2024-03-29T11:30:09.610 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.615+0000 7f669db32640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:09.610 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.615+0000 7f669db32640 10 monclient(hunting): authenticate will time out at 1433.933157s 2024-03-29T11:30:09.612 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.616+0000 7f6696ffd640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f66981422b0 0x7f669812e860 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:40778/0 (socket says 172.21.15.86:40778) 2024-03-29T11:30:09.612 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.616+0000 7f6696ffd640 1 -- 172.21.15.86:0/3554289540 learned_addr learned my addr 172.21.15.86:0/3554289540 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:30:09.612 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.617+0000 7f667ffff640 10 client.?.objecter ms_handle_connect 0x7f66981422b0 2024-03-29T11:30:09.612 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.617+0000 7f667ffff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:09.613 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.617+0000 7f667ffff640 1 -- 172.21.15.86:0/3554289540 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1517478588 0 0) 0x7f669806f160 con 0x7f66981422b0 2024-03-29T11:30:09.613 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.617+0000 7f667ffff640 10 monclient(hunting): _negotiate 2024-03-29T11:30:09.613 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.617+0000 7f667ffff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:09.613 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.617+0000 7f667ffff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:09.613 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.617+0000 7f667ffff640 10 monclient(hunting): my global_id is 4637 2024-03-29T11:30:09.613 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.617+0000 7f667ffff640 1 -- 172.21.15.86:0/3554289540 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6684004eb0 con 0x7f66981422b0 2024-03-29T11:30:09.613 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.618+0000 7f667ffff640 1 -- 172.21.15.86:0/3554289540 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2973038049 0 0) 0x7f6684004eb0 con 0x7f66981422b0 2024-03-29T11:30:09.613 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.618+0000 7f667ffff640 1 -- 172.21.15.86:0/3554289540 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f669806f160 con 0x7f66981422b0 2024-03-29T11:30:09.613 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.618+0000 7f667ffff640 1 -- 172.21.15.86:0/3554289540 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f66880017d0 con 0x7f66981422b0 2024-03-29T11:30:09.613 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.618+0000 7f667ffff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:09.613 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.618+0000 7f667ffff640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:09.613 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.618+0000 7f667ffff640 10 monclient(hunting): dump: 2024-03-29T11:30:09.613 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:30:09.613 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:09.613 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:09.614 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:09.614 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:09.614 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:09.614 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:30:09.614 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:30:09.614 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:30:09.614 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:09.614 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.618+0000 7f667ffff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:09.614 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.618+0000 7f667ffff640 1 -- 172.21.15.86:0/3554289540 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 251181997 0 0) 0x7f669806f160 con 0x7f66981422b0 2024-03-29T11:30:09.614 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.618+0000 7f667ffff640 1 -- 172.21.15.86:0/3554289540 >> v1:172.21.15.196:6789/0 conn(0x7f6698142690 legacy=0x7f669812ef70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:30:09.615 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.621+0000 7f667ffff640 1 -- 172.21.15.86:0/3554289540 >> v1:172.21.15.86:6789/0 conn(0x7f66980b17d0 legacy=0x7f66981deb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:09.616 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.621+0000 7f667ffff640 10 monclient: _finish_hunting 0 2024-03-29T11:30:09.616 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.621+0000 7f667ffff640 1 monclient: found mon.c 2024-03-29T11:30:09.616 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.621+0000 7f667ffff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:09.616 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.621+0000 7f667ffff640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:30:09.616 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.621+0000 7f667ffff640 1 -- 172.21.15.86:0/3554289540 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6698132ae0 con 0x7f66981422b0 2024-03-29T11:30:09.616 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.621+0000 7f667ffff640 10 monclient: _finish_auth 0 2024-03-29T11:30:09.616 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.622+0000 7f667ffff640 10 monclient: _check_auth_tickets 2024-03-29T11:30:09.616 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.622+0000 7f667ffff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:09.616 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.622+0000 7f669db32640 5 monclient: authenticate success, global_id 4637 2024-03-29T11:30:09.616 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.622+0000 7f669db32640 10 monclient: _renew_subs 2024-03-29T11:30:09.616 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.622+0000 7f669db32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:30:09.616 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.622+0000 7f669db32640 1 -- 172.21.15.86:0/3554289540 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6698132780 con 0x7f66981422b0 2024-03-29T11:30:09.616 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.622+0000 7f669db32640 10 client.4637.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:09.616 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.622+0000 7f669db32640 10 monclient: _renew_subs 2024-03-29T11:30:09.616 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.622+0000 7f669db32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:30:09.617 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.622+0000 7f669db32640 1 -- 172.21.15.86:0/3554289540 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f669806de90 con 0x7f66981422b0 2024-03-29T11:30:09.617 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.622+0000 7f669db32640 1 librados: init done 2024-03-29T11:30:09.617 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.622+0000 7f667ffff640 10 client.4637.objecter ms_handle_connect 0x7f66980b17d0 2024-03-29T11:30:09.617 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.622+0000 7f667ffff640 10 client.4637.objecter resend_mon_ops 2024-03-29T11:30:09.617 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.623+0000 7f669db32640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:09.617 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.623+0000 7f669db32640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:09.618 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.624+0000 7f669db32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:30:09.618 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.624+0000 7f669db32640 1 -- 172.21.15.86:0/3554289540 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f669806efa0 con 0x7f66981422b0 2024-03-29T11:30:09.618 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.624+0000 7f667ffff640 1 -- 172.21.15.86:0/3554289540 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f6688001930 con 0x7f66981422b0 2024-03-29T11:30:09.618 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.625+0000 7f667ffff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:09.618 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.625+0000 7f667ffff640 1 -- 172.21.15.86:0/3554289540 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f6688004a00 con 0x7f66981422b0 2024-03-29T11:30:09.618 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.625+0000 7f667ffff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:09.618 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.625+0000 7f667ffff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:09.618 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.625+0000 7f667ffff640 10 monclient: dump: 2024-03-29T11:30:09.618 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:30:09.618 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:09.618 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:09.618 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:09.618 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:09.619 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:09.619 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:30:09.619 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:30:09.619 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:30:09.619 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:09.619 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.625+0000 7f6695ffb640 4 set_mon_vals no callback set 2024-03-29T11:30:09.620 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.628+0000 7f667ffff640 1 -- 172.21.15.86:0/3554289540 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f6688004b60 con 0x7f66981422b0 2024-03-29T11:30:09.622 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.629+0000 7f66977fe640 10 monclient: get_auth_request con 0x7f668406dcf0 auth_method 0 2024-03-29T11:30:09.622 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.629+0000 7f667ffff640 1 -- 172.21.15.86:0/3554289540 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f6688084970 con 0x7f66981422b0 2024-03-29T11:30:09.622 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.629+0000 7f667ffff640 10 client.4637.objecter ms_dispatch 0x7f6698000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:30:09.622 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.629+0000 7f667ffff640 3 client.4637.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:30:09.622 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.629+0000 7f667ffff640 3 client.4637.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:30:09.622 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.629+0000 7f667ffff640 20 client.4637.objecter dump_active .. 0 homeless 2024-03-29T11:30:09.622 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.629+0000 7f667ffff640 10 client.4637.objecter ms_handle_connect 0x7f668406dcf0 2024-03-29T11:30:09.630 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.636+0000 7f667ffff640 1 -- 172.21.15.86:0/3554289540 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f6688050250 con 0x7f66981422b0 2024-03-29T11:30:09.630 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.636+0000 7f667ffff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:09.630 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.636+0000 7f667ffff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:09.958 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.966+0000 7f669db32640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:09.959 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.966+0000 7f669db32640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:09.959 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.966+0000 7f669db32640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:30:09.959 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.966+0000 7f669db32640 1 -- 172.21.15.86:0/3554289540 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6698075f10 con 0x7f66981422b0 2024-03-29T11:30:09.962 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.970+0000 7f667ffff640 1 -- 172.21.15.86:0/3554289540 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f6688052180 con 0x7f66981422b0 2024-03-29T11:30:09.962 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.970+0000 7f667ffff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:09.962 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.970+0000 7f667ffff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:09.962 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:30:09.962 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:09.965 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.973+0000 7f669db32640 10 librados: watch_flush enter 2024-03-29T11:30:09.966 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.973+0000 7f669db32640 10 librados: watch_flush exit 2024-03-29T11:30:09.966 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.973+0000 7f669db32640 20 client.4637.objecter shutdown clearing up homeless session... 2024-03-29T11:30:09.966 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.973+0000 7f669db32640 10 client.4637.objecter successfully canceled tick 2024-03-29T11:30:09.966 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.973+0000 7f669db32640 1 -- 172.21.15.86:0/3554289540 >> v1:172.21.15.196:6816/588110318 conn(0x7f668406dcf0 legacy=0x7f66840701b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:09.966 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.973+0000 7f669db32640 10 monclient: shutdown 2024-03-29T11:30:09.966 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.973+0000 7f669db32640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:09.966 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.973+0000 7f669db32640 1 -- 172.21.15.86:0/3554289540 >> v1:172.21.15.86:6790/0 conn(0x7f66981422b0 legacy=0x7f669812e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:09.966 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.973+0000 7f669db32640 1 -- 172.21.15.86:0/3554289540 shutdown_connections 2024-03-29T11:30:09.966 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.973+0000 7f669db32640 1 -- 172.21.15.86:0/3554289540 >> 172.21.15.86:0/3554289540 conn(0x7f6698066a00 msgr2=0x7f66980b2490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:09.967 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.975+0000 7f669db32640 1 -- 172.21.15.86:0/3554289540 shutdown_connections 2024-03-29T11:30:09.967 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.975+0000 7f669db32640 1 -- 172.21.15.86:0/3554289540 wait complete. 2024-03-29T11:30:09.968 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:09.975+0000 7f669db32640 1 librados: shutdown 2024-03-29T11:30:09.984 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:16.792 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:16.937 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.944+0000 7fac103b5640 10 monclient: get_monmap_and_config 2024-03-29T11:30:16.937 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.944+0000 7fac103b5640 10 monclient: build_initial_monmap 2024-03-29T11:30:16.937 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.944+0000 7fac103b5640 10 monclient: monmap: 2024-03-29T11:30:16.937 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:30:16.937 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:16.937 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:30:16.945331+0000 2024-03-29T11:30:16.937 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:30:16.945331+0000 2024-03-29T11:30:16.938 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:16.938 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:16.938 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:30:16.938 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:30:16.938 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:30:16.938 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:16.938 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.945+0000 7fac103b5640 1 Processor -- start 2024-03-29T11:30:16.938 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.945+0000 7fac103b5640 1 -- start start 2024-03-29T11:30:16.939 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.945+0000 7fac103b5640 10 monclient: init 2024-03-29T11:30:16.941 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.947+0000 7fac103b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:16.941 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.947+0000 7fac103b5640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:30:16.941 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.947+0000 7fac103b5640 10 monclient(hunting): picked mon.noname-c con 0x7fac080b4100 addr v1:172.21.15.196:6789/0 2024-03-29T11:30:16.941 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.947+0000 7fac103b5640 10 monclient(hunting): picked mon.noname-a con 0x7fac0813ee90 addr v1:172.21.15.86:6789/0 2024-03-29T11:30:16.942 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.947+0000 7fac103b5640 10 monclient(hunting): picked mon.noname-b con 0x7fac080af900 addr v1:172.21.15.86:6790/0 2024-03-29T11:30:16.942 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.947+0000 7fac103b5640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac080afd10 con 0x7fac0813ee90 2024-03-29T11:30:16.942 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.947+0000 7fac103b5640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac080af6f0 con 0x7fac080b4100 2024-03-29T11:30:16.942 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.947+0000 7fac103b5640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac080aaf60 con 0x7fac080af900 2024-03-29T11:30:16.942 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.947+0000 7fac103b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:16.942 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.947+0000 7fac103b5640 10 monclient(hunting): authenticate will time out at 1441.265155s 2024-03-29T11:30:16.942 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.948+0000 7fac0e952640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7fac080af900 0x7fac0813b5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:34470/0 (socket says 172.21.15.86:34470) 2024-03-29T11:30:16.942 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.948+0000 7fac0e952640 1 -- 172.21.15.86:0/2834704464 learned_addr learned my addr 172.21.15.86:0/2834704464 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:30:16.942 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.948+0000 7fac0d14f640 1 -- 172.21.15.86:0/2834704464 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1333641906 0 0) 0x7fac080aaf60 con 0x7fac080af900 2024-03-29T11:30:16.942 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.948+0000 7fac0d14f640 10 monclient(hunting): _negotiate 2024-03-29T11:30:16.942 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.948+0000 7fac0d14f640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:16.942 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.948+0000 7fac0d14f640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:16.943 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.948+0000 7fac0d14f640 10 monclient(hunting): my global_id is 4640 2024-03-29T11:30:16.943 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.948+0000 7fac0d14f640 1 -- 172.21.15.86:0/2834704464 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabf400bd30 con 0x7fac080af900 2024-03-29T11:30:16.943 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.948+0000 7fac0d14f640 1 -- 172.21.15.86:0/2834704464 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1598381225 0 0) 0x7fabf400bd30 con 0x7fac080af900 2024-03-29T11:30:16.943 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.948+0000 7fac0d14f640 1 -- 172.21.15.86:0/2834704464 >> v1:172.21.15.196:6789/0 conn(0x7fac080b4100 legacy=0x7fac080a9850 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:30:16.944 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.951+0000 7fac0d14f640 1 -- 172.21.15.86:0/2834704464 >> v1:172.21.15.86:6789/0 conn(0x7fac0813ee90 legacy=0x7fac080adfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:30:16.944 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.951+0000 7fac0d14f640 10 monclient: _finish_hunting 0 2024-03-29T11:30:16.944 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.951+0000 7fac0d14f640 1 monclient: found mon.noname-b 2024-03-29T11:30:16.944 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.951+0000 7fac0d14f640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:16.944 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.951+0000 7fac0d14f640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:30:16.944 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.951+0000 7fac0d14f640 1 -- 172.21.15.86:0/2834704464 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac0813e970 con 0x7fac080af900 2024-03-29T11:30:16.944 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.951+0000 7fac0d14f640 10 monclient: _finish_auth 0 2024-03-29T11:30:16.944 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.951+0000 7fac0d14f640 10 monclient: _check_auth_tickets 2024-03-29T11:30:16.944 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.951+0000 7fac0d14f640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:16.945 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.951+0000 7fac103b5640 5 monclient: authenticate success, global_id 4640 2024-03-29T11:30:16.945 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.951+0000 7fac103b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:16.945 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.951+0000 7fac0d14f640 1 -- 172.21.15.86:0/2834704464 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fabf8001690 con 0x7fac080af900 2024-03-29T11:30:16.945 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.951+0000 7fac0d14f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:16.945 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.951+0000 7fac0d14f640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:16.945 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.951+0000 7fac0d14f640 10 monclient: dump: 2024-03-29T11:30:16.945 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:30:16.945 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:16.945 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:16.945 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:16.945 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:16.945 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:16.945 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:30:16.945 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:30:16.946 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:30:16.946 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:16.946 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.951+0000 7fac103b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:16.946 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.952+0000 7fac0d14f640 1 -- 172.21.15.86:0/2834704464 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fabf8001860 con 0x7fac080af900 2024-03-29T11:30:16.946 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.952+0000 7fac0d14f640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:16.946 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.952+0000 7fac103b5640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:16.946 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.952+0000 7fac103b5640 4 set_mon_vals no callback set 2024-03-29T11:30:16.947 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.952+0000 7fac0d14f640 1 -- 172.21.15.86:0/2834704464 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fabf8001cb0 con 0x7fac080af900 2024-03-29T11:30:16.947 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.952+0000 7fac0d14f640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:16.947 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.952+0000 7fac0d14f640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:16.947 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.952+0000 7fac0d14f640 10 monclient: dump: 2024-03-29T11:30:16.948 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:30:16.948 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:16.948 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:16.948 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:16.948 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:16.948 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:16.948 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:30:16.948 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:30:16.948 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:30:16.948 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:16.948 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.953+0000 7fac103b5640 10 monclient: shutdown 2024-03-29T11:30:16.948 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.953+0000 7fac103b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:16.949 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.953+0000 7fac103b5640 1 -- 172.21.15.86:0/2834704464 >> v1:172.21.15.86:6790/0 conn(0x7fac080af900 legacy=0x7fac0813b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:16.949 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.953+0000 7fac103b5640 1 -- 172.21.15.86:0/2834704464 shutdown_connections 2024-03-29T11:30:16.949 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.953+0000 7fac103b5640 1 -- 172.21.15.86:0/2834704464 >> 172.21.15.86:0/2834704464 conn(0x7fac080b35e0 msgr2=0x7fac080b39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:16.949 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.953+0000 7fac103b5640 1 -- 172.21.15.86:0/2834704464 shutdown_connections 2024-03-29T11:30:16.949 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.953+0000 7fac103b5640 1 -- 172.21.15.86:0/2834704464 wait complete. 2024-03-29T11:30:16.949 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.954+0000 7fac103b5640 10 monclient: build_initial_monmap 2024-03-29T11:30:16.949 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.954+0000 7fac103b5640 10 monclient: monmap: 2024-03-29T11:30:16.949 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:30:16.949 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:16.949 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:30:16.949 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:30:16.949 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:16.949 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:16.949 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:30:16.949 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:30:16.950 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:30:16.950 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:16.950 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.954+0000 7fac103b5640 1 librados: starting msgr at 2024-03-29T11:30:16.950 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.954+0000 7fac103b5640 1 librados: starting objecter 2024-03-29T11:30:16.950 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.955+0000 7fac103b5640 1 Processor -- start 2024-03-29T11:30:16.954 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.955+0000 7fac103b5640 1 -- start start 2024-03-29T11:30:16.954 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.955+0000 7fac103b5640 1 librados: setting wanted keys 2024-03-29T11:30:16.954 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.955+0000 7fac103b5640 1 librados: calling monclient init 2024-03-29T11:30:16.954 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.955+0000 7fac103b5640 10 monclient: init 2024-03-29T11:30:16.954 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.955+0000 7fac103b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:16.954 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.955+0000 7fac103b5640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:30:16.954 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.956+0000 7fac103b5640 10 monclient(hunting): picked mon.noname-c con 0x7fac080af900 addr v1:172.21.15.86:6790/0 2024-03-29T11:30:16.954 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.956+0000 7fac103b5640 10 monclient(hunting): picked mon.noname-a con 0x7fac080b4100 addr v1:172.21.15.86:6789/0 2024-03-29T11:30:16.954 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.956+0000 7fac103b5640 10 monclient(hunting): picked mon.noname-b con 0x7fac0813ee90 addr v1:172.21.15.196:6789/0 2024-03-29T11:30:16.954 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.956+0000 7fac103b5640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac0806e9f0 con 0x7fac080b4100 2024-03-29T11:30:16.954 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.956+0000 7fac103b5640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac081d2fd0 con 0x7fac0813ee90 2024-03-29T11:30:16.955 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.956+0000 7fac103b5640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac0813caa0 con 0x7fac080af900 2024-03-29T11:30:16.955 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.956+0000 7fac103b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:16.955 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.956+0000 7fac103b5640 10 monclient(hunting): authenticate will time out at 1441.274063s 2024-03-29T11:30:16.955 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.956+0000 7fac0e151640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7fac080af900 0x7fac08141080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:34902/0 (socket says 172.21.15.86:34902) 2024-03-29T11:30:16.955 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.956+0000 7fac0e151640 1 -- 172.21.15.86:0/1200833597 learned_addr learned my addr 172.21.15.86:0/1200833597 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:30:16.955 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.957+0000 7fabfeffd640 10 client.?.objecter ms_handle_connect 0x7fac080af900 2024-03-29T11:30:16.955 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.957+0000 7fabfeffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:16.955 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.957+0000 7fabfeffd640 1 -- 172.21.15.86:0/1200833597 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 334249154 0 0) 0x7fac0813caa0 con 0x7fac080af900 2024-03-29T11:30:16.955 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.957+0000 7fabfeffd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:16.955 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.957+0000 7fabfeffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:16.955 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.957+0000 7fabfeffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:16.955 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.957+0000 7fabfeffd640 10 monclient(hunting): my global_id is 4643 2024-03-29T11:30:16.955 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.957+0000 7fabfeffd640 1 -- 172.21.15.86:0/1200833597 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabec004eb0 con 0x7fac080af900 2024-03-29T11:30:16.955 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.958+0000 7fabfeffd640 1 -- 172.21.15.86:0/1200833597 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2066352853 0 0) 0x7fabec004eb0 con 0x7fac080af900 2024-03-29T11:30:16.955 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.958+0000 7fabfeffd640 1 -- 172.21.15.86:0/1200833597 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fac0813caa0 con 0x7fac080af900 2024-03-29T11:30:16.956 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.958+0000 7fabfeffd640 1 -- 172.21.15.86:0/1200833597 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fac00001bf0 con 0x7fac080af900 2024-03-29T11:30:16.956 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.958+0000 7fabfeffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:16.956 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.958+0000 7fabfeffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:16.956 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.958+0000 7fabfeffd640 10 monclient(hunting): dump: 2024-03-29T11:30:16.956 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:30:16.956 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:16.956 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:16.956 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:16.956 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:16.956 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:16.956 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:30:16.956 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:30:16.956 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:30:16.956 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:16.957 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.958+0000 7fabfeffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:16.957 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.959+0000 7fabfeffd640 1 -- 172.21.15.86:0/1200833597 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 2439327558 0 0) 0x7fac0813caa0 con 0x7fac080af900 2024-03-29T11:30:16.957 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.959+0000 7fabfeffd640 1 -- 172.21.15.86:0/1200833597 >> v1:172.21.15.196:6789/0 conn(0x7fac0813ee90 legacy=0x7fac081358b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:30:16.961 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.962+0000 7fabfeffd640 1 -- 172.21.15.86:0/1200833597 >> v1:172.21.15.86:6789/0 conn(0x7fac080b4100 legacy=0x7fac081d34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:16.961 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.962+0000 7fabfeffd640 10 monclient: _finish_hunting 0 2024-03-29T11:30:16.961 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.962+0000 7fabfeffd640 1 monclient: found mon.c 2024-03-29T11:30:16.961 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.962+0000 7fabfeffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:16.961 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.962+0000 7fabfeffd640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:30:16.961 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.962+0000 7fabfeffd640 1 -- 172.21.15.86:0/1200833597 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac0813cc30 con 0x7fac080af900 2024-03-29T11:30:16.961 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.962+0000 7fabfeffd640 10 monclient: _finish_auth 0 2024-03-29T11:30:16.961 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.962+0000 7fabfeffd640 10 monclient: _check_auth_tickets 2024-03-29T11:30:16.961 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.962+0000 7fabfeffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:16.961 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.962+0000 7fac103b5640 5 monclient: authenticate success, global_id 4643 2024-03-29T11:30:16.961 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.962+0000 7fac103b5640 10 monclient: _renew_subs 2024-03-29T11:30:16.961 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.962+0000 7fac103b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:30:16.961 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.962+0000 7fac103b5640 1 -- 172.21.15.86:0/1200833597 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac081417f0 con 0x7fac080af900 2024-03-29T11:30:16.962 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.963+0000 7fac103b5640 10 client.4643.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:16.962 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.963+0000 7fac103b5640 10 monclient: _renew_subs 2024-03-29T11:30:16.962 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.963+0000 7fac103b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:30:16.962 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.963+0000 7fac103b5640 1 -- 172.21.15.86:0/1200833597 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fac081d3de0 con 0x7fac080af900 2024-03-29T11:30:16.962 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.963+0000 7fac103b5640 1 librados: init done 2024-03-29T11:30:16.962 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.963+0000 7fabfeffd640 1 -- 172.21.15.86:0/1200833597 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fac00001d50 con 0x7fac080af900 2024-03-29T11:30:16.962 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.963+0000 7fabfeffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:16.962 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.963+0000 7fabfeffd640 1 -- 172.21.15.86:0/1200833597 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fac00004cc0 con 0x7fac080af900 2024-03-29T11:30:16.962 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.963+0000 7fac0c94e640 4 set_mon_vals no callback set 2024-03-29T11:30:16.962 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.964+0000 7fabfeffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:16.962 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.964+0000 7fabfeffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:16.962 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.964+0000 7fac103b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:16.962 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.964+0000 7fabfeffd640 10 monclient: dump: 2024-03-29T11:30:16.962 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:30:16.963 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:16.963 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:16.963 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:16.963 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:16.963 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:16.963 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:30:16.963 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:30:16.963 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:30:16.963 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:16.963 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.964+0000 7fac103b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:16.963 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.965+0000 7fac103b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:30:16.963 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.965+0000 7fac103b5640 1 -- 172.21.15.86:0/1200833597 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabf0005080 con 0x7fac080af900 2024-03-29T11:30:16.963 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.966+0000 7fabfeffd640 1 -- 172.21.15.86:0/1200833597 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7fac00004e20 con 0x7fac080af900 2024-03-29T11:30:16.963 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.967+0000 7fabfeffd640 1 -- 172.21.15.86:0/1200833597 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7fac00084cf0 con 0x7fac080af900 2024-03-29T11:30:16.964 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.967+0000 7fabfeffd640 10 client.4643.objecter ms_dispatch 0x7fac08000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:30:16.964 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.967+0000 7fabfeffd640 3 client.4643.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:30:16.964 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.968+0000 7fabfeffd640 3 client.4643.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:30:16.964 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.968+0000 7fabfeffd640 20 client.4643.objecter dump_active .. 0 homeless 2024-03-29T11:30:16.964 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.968+0000 7fac0d950640 10 monclient: get_auth_request con 0x7fabec06dc60 auth_method 0 2024-03-29T11:30:16.964 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.968+0000 7fabfeffd640 10 client.4643.objecter ms_handle_connect 0x7fabec06dc60 2024-03-29T11:30:16.967 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.974+0000 7fabfeffd640 1 -- 172.21.15.86:0/1200833597 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fac00050400 con 0x7fac080af900 2024-03-29T11:30:16.969 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.977+0000 7fabfeffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:16.969 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:16.977+0000 7fabfeffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:17.285 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:17.292+0000 7fac103b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:17.285 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:17.292+0000 7fac103b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:17.285 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:17.292+0000 7fac103b5640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:30:17.285 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:17.292+0000 7fac103b5640 1 -- 172.21.15.86:0/1200833597 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fabf0005330 con 0x7fac080af900 2024-03-29T11:30:17.285 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:17.293+0000 7fabfeffd640 1 -- 172.21.15.86:0/1200833597 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7fac00052500 con 0x7fac080af900 2024-03-29T11:30:17.285 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:17.293+0000 7fabfeffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:17.285 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:17.293+0000 7fabfeffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:17.286 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:30:17.286 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:17.288 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:17.296+0000 7fac103b5640 10 librados: watch_flush enter 2024-03-29T11:30:17.291 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:17.299+0000 7fac103b5640 10 librados: watch_flush exit 2024-03-29T11:30:17.291 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:17.299+0000 7fac103b5640 20 client.4643.objecter shutdown clearing up homeless session... 2024-03-29T11:30:17.291 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:17.299+0000 7fac103b5640 10 client.4643.objecter successfully canceled tick 2024-03-29T11:30:17.292 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:17.299+0000 7fac103b5640 1 -- 172.21.15.86:0/1200833597 >> v1:172.21.15.196:6816/588110318 conn(0x7fabec06dc60 legacy=0x7fabec070120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:17.292 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:17.299+0000 7fac103b5640 10 monclient: shutdown 2024-03-29T11:30:17.292 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:17.299+0000 7fac103b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:17.292 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:17.299+0000 7fac103b5640 1 -- 172.21.15.86:0/1200833597 >> v1:172.21.15.86:6790/0 conn(0x7fac080af900 legacy=0x7fac08141080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:17.293 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:17.299+0000 7fac103b5640 1 -- 172.21.15.86:0/1200833597 shutdown_connections 2024-03-29T11:30:17.293 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:17.299+0000 7fac103b5640 1 -- 172.21.15.86:0/1200833597 >> 172.21.15.86:0/1200833597 conn(0x7fac080b35e0 msgr2=0x7fac081431d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:17.293 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:17.300+0000 7fac103b5640 1 -- 172.21.15.86:0/1200833597 shutdown_connections 2024-03-29T11:30:17.293 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:17.300+0000 7fac103b5640 1 -- 172.21.15.86:0/1200833597 wait complete. 2024-03-29T11:30:17.293 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:17.300+0000 7fac103b5640 1 librados: shutdown 2024-03-29T11:30:17.310 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:24.117 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:24.267 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.273+0000 7f72eb6c8640 10 monclient: get_monmap_and_config 2024-03-29T11:30:24.267 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.273+0000 7f72eb6c8640 10 monclient: build_initial_monmap 2024-03-29T11:30:24.267 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.273+0000 7f72eb6c8640 10 monclient: monmap: 2024-03-29T11:30:24.267 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:30:24.267 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:24.267 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:30:24.274774+0000 2024-03-29T11:30:24.267 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:30:24.274774+0000 2024-03-29T11:30:24.267 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:24.268 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:24.268 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:30:24.268 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:30:24.268 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:30:24.268 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:24.268 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.274+0000 7f72eb6c8640 1 Processor -- start 2024-03-29T11:30:24.268 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.274+0000 7f72eb6c8640 1 -- start start 2024-03-29T11:30:24.270 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.274+0000 7f72eb6c8640 10 monclient: init 2024-03-29T11:30:24.270 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.275+0000 7f72eb6c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:24.270 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.275+0000 7f72eb6c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:30:24.270 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.275+0000 7f72eb6c8640 10 monclient(hunting): picked mon.noname-b con 0x7f72e40a9750 addr v1:172.21.15.86:6790/0 2024-03-29T11:30:24.270 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.275+0000 7f72eb6c8640 10 monclient(hunting): picked mon.noname-c con 0x7f72e413e1e0 addr v1:172.21.15.196:6789/0 2024-03-29T11:30:24.270 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.275+0000 7f72eb6c8640 10 monclient(hunting): picked mon.noname-a con 0x7f72e4071c30 addr v1:172.21.15.86:6789/0 2024-03-29T11:30:24.270 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.275+0000 7f72eb6c8640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e40ab300 con 0x7f72e4071c30 2024-03-29T11:30:24.271 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.275+0000 7f72eb6c8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e40ac2f0 con 0x7f72e413e1e0 2024-03-29T11:30:24.271 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.275+0000 7f72eb6c8640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e40ad8d0 con 0x7f72e40a9750 2024-03-29T11:30:24.271 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.275+0000 7f72eb6c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:24.271 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.275+0000 7f72eb6c8640 10 monclient(hunting): authenticate will time out at 1448.593976s 2024-03-29T11:30:24.271 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.276+0000 7f72e9c65640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f72e4071c30 0x7f72e40ace20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:56538/0 (socket says 172.21.15.86:56538) 2024-03-29T11:30:24.271 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.276+0000 7f72e9c65640 1 -- 172.21.15.86:0/440243318 learned_addr learned my addr 172.21.15.86:0/440243318 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:30:24.271 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.276+0000 7f72dbfff640 1 -- 172.21.15.86:0/440243318 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 217404249 0 0) 0x7f72e40ab300 con 0x7f72e4071c30 2024-03-29T11:30:24.271 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.276+0000 7f72dbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:30:24.271 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.276+0000 7f72dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:24.271 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.276+0000 7f72dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:24.271 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.276+0000 7f72dbfff640 10 monclient(hunting): my global_id is 4545 2024-03-29T11:30:24.271 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.276+0000 7f72dbfff640 1 -- 172.21.15.86:0/440243318 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72cc003520 con 0x7f72e4071c30 2024-03-29T11:30:24.271 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.277+0000 7f72dbfff640 1 -- 172.21.15.86:0/440243318 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 13533221 0 0) 0x7f72e40ad8d0 con 0x7f72e40a9750 2024-03-29T11:30:24.271 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.277+0000 7f72dbfff640 10 monclient(hunting): _negotiate 2024-03-29T11:30:24.272 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.277+0000 7f72dbfff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:24.272 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.277+0000 7f72dbfff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:24.272 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.278+0000 7f72dbfff640 10 monclient(hunting): my global_id is 4646 2024-03-29T11:30:24.275 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.278+0000 7f72dbfff640 1 -- 172.21.15.86:0/440243318 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72e40ab300 con 0x7f72e40a9750 2024-03-29T11:30:24.275 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.278+0000 7f72dbfff640 1 -- 172.21.15.86:0/440243318 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 890579305 0 0) 0x7f72cc003520 con 0x7f72e4071c30 2024-03-29T11:30:24.276 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.278+0000 7f72dbfff640 1 -- 172.21.15.86:0/440243318 >> v1:172.21.15.86:6790/0 conn(0x7f72e40a9750 legacy=0x7f72e413dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:24.276 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.278+0000 7f72dbfff640 1 -- 172.21.15.86:0/440243318 >> v1:172.21.15.196:6789/0 conn(0x7f72e413e1e0 legacy=0x7f72e40ac710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:24.276 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.278+0000 7f72dbfff640 10 monclient: _finish_hunting 0 2024-03-29T11:30:24.276 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.278+0000 7f72dbfff640 1 monclient: found mon.noname-a 2024-03-29T11:30:24.276 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.279+0000 7f72dbfff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:24.276 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.279+0000 7f72dbfff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.86:6789/0 2024-03-29T11:30:24.276 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.279+0000 7f72dbfff640 1 -- 172.21.15.86:0/440243318 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e40ada60 con 0x7f72e4071c30 2024-03-29T11:30:24.276 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.279+0000 7f72dbfff640 10 monclient: _finish_auth 0 2024-03-29T11:30:24.276 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.279+0000 7f72dbfff640 10 monclient: _check_auth_tickets 2024-03-29T11:30:24.276 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.279+0000 7f72dbfff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:24.276 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.279+0000 7f72dbfff640 1 -- 172.21.15.86:0/440243318 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f72dc001690 con 0x7f72e4071c30 2024-03-29T11:30:24.276 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.279+0000 7f72eb6c8640 5 monclient: authenticate success, global_id 4545 2024-03-29T11:30:24.276 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.279+0000 7f72eb6c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:24.277 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.279+0000 7f72dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:24.277 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.279+0000 7f72dbfff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:24.277 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.279+0000 7f72dbfff640 10 monclient: dump: 2024-03-29T11:30:24.277 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:30:24.277 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:24.277 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:24.277 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:24.277 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:24.277 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:24.277 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:30:24.277 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:30:24.277 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:30:24.277 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:24.277 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.279+0000 7f72dbfff640 1 -- 172.21.15.86:0/440243318 <== mon.0 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72dc0018b0 con 0x7f72e4071c30 2024-03-29T11:30:24.278 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.279+0000 7f72eb6c8640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:24.278 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.279+0000 7f72dbfff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:24.278 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.279+0000 7f72dbfff640 1 -- 172.21.15.86:0/440243318 <== mon.0 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f72dc001c40 con 0x7f72e4071c30 2024-03-29T11:30:24.278 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.280+0000 7f72eb6c8640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:24.278 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.280+0000 7f72eb6c8640 4 set_mon_vals no callback set 2024-03-29T11:30:24.278 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.280+0000 7f72dbfff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:24.278 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.280+0000 7f72dbfff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:24.278 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.280+0000 7f72dbfff640 10 monclient: dump: 2024-03-29T11:30:24.278 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:30:24.278 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:24.278 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:24.278 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:24.278 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:24.278 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:24.279 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:30:24.279 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:30:24.279 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:30:24.279 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:24.279 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.280+0000 7f72eb6c8640 10 monclient: shutdown 2024-03-29T11:30:24.279 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.280+0000 7f72eb6c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:24.279 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.280+0000 7f72eb6c8640 1 -- 172.21.15.86:0/440243318 >> v1:172.21.15.86:6789/0 conn(0x7f72e4071c30 legacy=0x7f72e40ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:24.279 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.280+0000 7f72eb6c8640 1 -- 172.21.15.86:0/440243318 shutdown_connections 2024-03-29T11:30:24.279 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.281+0000 7f72eb6c8640 1 -- 172.21.15.86:0/440243318 >> 172.21.15.86:0/440243318 conn(0x7f72e406b660 msgr2=0x7f72e406ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:24.279 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.281+0000 7f72eb6c8640 1 -- 172.21.15.86:0/440243318 shutdown_connections 2024-03-29T11:30:24.279 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.281+0000 7f72eb6c8640 1 -- 172.21.15.86:0/440243318 wait complete. 2024-03-29T11:30:24.279 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.282+0000 7f72eb6c8640 10 monclient: build_initial_monmap 2024-03-29T11:30:24.279 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.282+0000 7f72eb6c8640 10 monclient: monmap: 2024-03-29T11:30:24.279 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:30:24.279 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:24.280 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:30:24.280 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:30:24.280 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:24.280 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:24.280 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:30:24.280 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:30:24.280 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:30:24.280 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:24.280 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.282+0000 7f72eb6c8640 1 librados: starting msgr at 2024-03-29T11:30:24.280 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.282+0000 7f72eb6c8640 1 librados: starting objecter 2024-03-29T11:30:24.280 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.282+0000 7f72eb6c8640 1 Processor -- start 2024-03-29T11:30:24.280 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.282+0000 7f72eb6c8640 1 -- start start 2024-03-29T11:30:24.280 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.282+0000 7f72eb6c8640 1 librados: setting wanted keys 2024-03-29T11:30:24.280 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.282+0000 7f72eb6c8640 1 librados: calling monclient init 2024-03-29T11:30:24.280 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.282+0000 7f72eb6c8640 10 monclient: init 2024-03-29T11:30:24.281 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.283+0000 7f72eb6c8640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:24.281 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.283+0000 7f72eb6c8640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:30:24.281 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.283+0000 7f72eb6c8640 10 monclient(hunting): picked mon.noname-c con 0x7f72e4071c30 addr v1:172.21.15.86:6790/0 2024-03-29T11:30:24.284 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.283+0000 7f72eb6c8640 10 monclient(hunting): picked mon.noname-b con 0x7f72e40a9750 addr v1:172.21.15.196:6789/0 2024-03-29T11:30:24.284 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.283+0000 7f72eb6c8640 10 monclient(hunting): picked mon.noname-a con 0x7f72e413e1e0 addr v1:172.21.15.86:6789/0 2024-03-29T11:30:24.284 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.283+0000 7f72eb6c8640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e41418a0 con 0x7f72e413e1e0 2024-03-29T11:30:24.285 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.283+0000 7f72eb6c8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e40713b0 con 0x7f72e40a9750 2024-03-29T11:30:24.285 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.283+0000 7f72eb6c8640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72e4071620 con 0x7f72e4071c30 2024-03-29T11:30:24.285 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.283+0000 7f72eb6c8640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:24.285 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.283+0000 7f72eb6c8640 10 monclient(hunting): authenticate will time out at 1448.601830s 2024-03-29T11:30:24.285 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.283+0000 7f72e9c65640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f72e413e1e0 0x7f72e41e20e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:56972/0 (socket says 172.21.15.86:56972) 2024-03-29T11:30:24.285 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.283+0000 7f72e9c65640 1 -- 172.21.15.86:0/3929770378 learned_addr learned my addr 172.21.15.86:0/3929770378 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:30:24.285 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.284+0000 7f72d9ffb640 10 client.?.objecter ms_handle_connect 0x7f72e413e1e0 2024-03-29T11:30:24.285 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.284+0000 7f72d9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:24.285 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.284+0000 7f72d9ffb640 10 client.?.objecter ms_handle_connect 0x7f72e40a9750 2024-03-29T11:30:24.285 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.284+0000 7f72d9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:24.285 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.284+0000 7f72d9ffb640 10 client.?.objecter ms_handle_connect 0x7f72e4071c30 2024-03-29T11:30:24.285 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.284+0000 7f72d9ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:24.285 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.284+0000 7f72d9ffb640 1 -- 172.21.15.86:0/3929770378 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 775898259 0 0) 0x7f72e41418a0 con 0x7f72e413e1e0 2024-03-29T11:30:24.285 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.284+0000 7f72d9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:30:24.286 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.284+0000 7f72d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:24.286 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.284+0000 7f72d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:24.286 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.284+0000 7f72d9ffb640 10 monclient(hunting): my global_id is 4548 2024-03-29T11:30:24.286 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.284+0000 7f72d9ffb640 1 -- 172.21.15.86:0/3929770378 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72d0004eb0 con 0x7f72e413e1e0 2024-03-29T11:30:24.286 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.284+0000 7f72d9ffb640 1 -- 172.21.15.86:0/3929770378 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 648109128 0 0) 0x7f72e40713b0 con 0x7f72e40a9750 2024-03-29T11:30:24.286 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.284+0000 7f72d9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:30:24.286 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.284+0000 7f72d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:24.286 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.284+0000 7f72d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:24.286 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.284+0000 7f72d9ffb640 10 monclient(hunting): my global_id is 4522 2024-03-29T11:30:24.286 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.284+0000 7f72d9ffb640 1 -- 172.21.15.86:0/3929770378 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72e41418a0 con 0x7f72e40a9750 2024-03-29T11:30:24.286 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.284+0000 7f72d9ffb640 1 -- 172.21.15.86:0/3929770378 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2431837957 0 0) 0x7f72e4071620 con 0x7f72e4071c30 2024-03-29T11:30:24.286 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.284+0000 7f72d9ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:30:24.286 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.284+0000 7f72d9ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:24.286 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.284+0000 7f72d9ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:24.286 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.284+0000 7f72d9ffb640 10 monclient(hunting): my global_id is 4649 2024-03-29T11:30:24.287 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.284+0000 7f72d9ffb640 1 -- 172.21.15.86:0/3929770378 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f72e40713b0 con 0x7f72e4071c30 2024-03-29T11:30:24.287 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.284+0000 7f72d9ffb640 1 -- 172.21.15.86:0/3929770378 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 780533982 0 0) 0x7f72d0004eb0 con 0x7f72e413e1e0 2024-03-29T11:30:24.287 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.285+0000 7f72d9ffb640 1 -- 172.21.15.86:0/3929770378 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72e4071620 con 0x7f72e413e1e0 2024-03-29T11:30:24.287 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.285+0000 7f72d9ffb640 1 -- 172.21.15.86:0/3929770378 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f72dc001bc0 con 0x7f72e413e1e0 2024-03-29T11:30:24.287 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.285+0000 7f72d9ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:24.287 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.285+0000 7f72d9ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:24.287 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.285+0000 7f72d9ffb640 10 monclient(hunting): dump: 2024-03-29T11:30:24.287 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:30:24.287 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:24.287 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:24.287 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:24.287 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:24.287 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:24.287 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:30:24.287 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:30:24.288 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:30:24.288 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:24.288 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.285+0000 7f72d9ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:24.288 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.285+0000 7f72d9ffb640 1 -- 172.21.15.86:0/3929770378 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 546049293 0 0) 0x7f72e40713b0 con 0x7f72e4071c30 2024-03-29T11:30:24.288 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.285+0000 7f72d9ffb640 1 -- 172.21.15.86:0/3929770378 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f72d0004eb0 con 0x7f72e4071c30 2024-03-29T11:30:24.288 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.285+0000 7f72d9ffb640 1 -- 172.21.15.86:0/3929770378 <== mon.0 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3394527032 0 0) 0x7f72e4071620 con 0x7f72e413e1e0 2024-03-29T11:30:24.288 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.285+0000 7f72d9ffb640 1 -- 172.21.15.86:0/3929770378 >> v1:172.21.15.86:6790/0 conn(0x7f72e4071c30 legacy=0x7f72e41422a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:24.288 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.285+0000 7f72d9ffb640 1 -- 172.21.15.86:0/3929770378 >> v1:172.21.15.196:6789/0 conn(0x7f72e40a9750 legacy=0x7f72e413fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:24.288 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.285+0000 7f72d9ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:30:24.288 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.285+0000 7f72d9ffb640 1 monclient: found mon.a 2024-03-29T11:30:24.288 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.285+0000 7f72d9ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:24.288 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.285+0000 7f72d9ffb640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:30:24.288 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.285+0000 7f72d9ffb640 1 -- 172.21.15.86:0/3929770378 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e41429b0 con 0x7f72e413e1e0 2024-03-29T11:30:24.288 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.285+0000 7f72d9ffb640 10 monclient: _finish_auth 0 2024-03-29T11:30:24.288 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.285+0000 7f72d9ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:30:24.289 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.285+0000 7f72d9ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:24.289 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.285+0000 7f72eb6c8640 5 monclient: authenticate success, global_id 4548 2024-03-29T11:30:24.289 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.285+0000 7f72eb6c8640 10 monclient: _renew_subs 2024-03-29T11:30:24.289 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.285+0000 7f72eb6c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:30:24.289 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.285+0000 7f72eb6c8640 1 -- 172.21.15.86:0/3929770378 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72e4140760 con 0x7f72e413e1e0 2024-03-29T11:30:24.289 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.285+0000 7f72eb6c8640 10 client.4548.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:24.289 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.285+0000 7f72eb6c8640 10 monclient: _renew_subs 2024-03-29T11:30:24.289 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.285+0000 7f72eb6c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:30:24.289 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.285+0000 7f72eb6c8640 1 -- 172.21.15.86:0/3929770378 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f72e413fd90 con 0x7f72e413e1e0 2024-03-29T11:30:24.289 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.285+0000 7f72eb6c8640 1 librados: init done 2024-03-29T11:30:24.289 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.286+0000 7f72d9ffb640 1 -- 172.21.15.86:0/3929770378 <== mon.0 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f72dc004a90 con 0x7f72e413e1e0 2024-03-29T11:30:24.289 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.286+0000 7f72d9ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:24.289 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.286+0000 7f72d9ffb640 1 -- 172.21.15.86:0/3929770378 <== mon.0 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f72dc001d20 con 0x7f72e413e1e0 2024-03-29T11:30:24.289 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.286+0000 7f72d9ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:24.289 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.286+0000 7f72d9ffb640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:24.290 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.286+0000 7f72d9ffb640 10 monclient: dump: 2024-03-29T11:30:24.290 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:30:24.290 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:24.290 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:24.290 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:24.290 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:24.290 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:24.290 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:30:24.290 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:30:24.290 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:30:24.290 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:24.290 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.286+0000 7f72db7fe640 4 set_mon_vals no callback set 2024-03-29T11:30:24.290 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.287+0000 7f72eb6c8640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:24.290 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.288+0000 7f72eb6c8640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:24.290 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.288+0000 7f72eb6c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:30:24.291 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.288+0000 7f72eb6c8640 1 -- 172.21.15.86:0/3929770378 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72c4005080 con 0x7f72e413e1e0 2024-03-29T11:30:24.291 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.288+0000 7f72d9ffb640 1 -- 172.21.15.86:0/3929770378 <== mon.0 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f72dc01ac10 con 0x7f72e413e1e0 2024-03-29T11:30:24.291 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.289+0000 7f72d9ffb640 1 -- 172.21.15.86:0/3929770378 <== mon.0 v1:172.21.15.86:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7f72dc084b00 con 0x7f72e413e1e0 2024-03-29T11:30:24.291 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.289+0000 7f72d9ffb640 10 client.4548.objecter ms_dispatch 0x7f72e4000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:30:24.291 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.289+0000 7f72d9ffb640 3 client.4548.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:30:24.291 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.289+0000 7f72d9ffb640 3 client.4548.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:30:24.291 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.289+0000 7f72d9ffb640 20 client.4548.objecter dump_active .. 0 homeless 2024-03-29T11:30:24.291 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.289+0000 7f72e9464640 10 monclient: get_auth_request con 0x7f72d006df00 auth_method 0 2024-03-29T11:30:24.291 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.289+0000 7f72d9ffb640 10 client.4548.objecter ms_handle_connect 0x7f72d006df00 2024-03-29T11:30:24.292 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.299+0000 7f72d9ffb640 1 -- 172.21.15.86:0/3929770378 <== mon.0 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f72dc0503e0 con 0x7f72e413e1e0 2024-03-29T11:30:24.292 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.299+0000 7f72d9ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:24.292 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.299+0000 7f72d9ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:24.683 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.690+0000 7f72eb6c8640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:24.684 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.690+0000 7f72eb6c8640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:24.684 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.690+0000 7f72eb6c8640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:30:24.684 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.690+0000 7f72eb6c8640 1 -- 172.21.15.86:0/3929770378 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f72c4005330 con 0x7f72e413e1e0 2024-03-29T11:30:24.684 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.692+0000 7f72d9ffb640 1 -- 172.21.15.86:0/3929770378 <== mon.0 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7f72dc052310 con 0x7f72e413e1e0 2024-03-29T11:30:24.684 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.692+0000 7f72d9ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:24.685 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.692+0000 7f72d9ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:24.685 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:30:24.685 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:24.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.695+0000 7f72eb6c8640 10 librados: watch_flush enter 2024-03-29T11:30:24.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.695+0000 7f72eb6c8640 10 librados: watch_flush exit 2024-03-29T11:30:24.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.695+0000 7f72eb6c8640 20 client.4548.objecter shutdown clearing up homeless session... 2024-03-29T11:30:24.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.695+0000 7f72eb6c8640 10 client.4548.objecter successfully canceled tick 2024-03-29T11:30:24.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.695+0000 7f72eb6c8640 1 -- 172.21.15.86:0/3929770378 >> v1:172.21.15.196:6816/588110318 conn(0x7f72d006df00 legacy=0x7f72d00703c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:24.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.695+0000 7f72eb6c8640 10 monclient: shutdown 2024-03-29T11:30:24.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.695+0000 7f72eb6c8640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:24.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.695+0000 7f72eb6c8640 1 -- 172.21.15.86:0/3929770378 >> v1:172.21.15.86:6789/0 conn(0x7f72e413e1e0 legacy=0x7f72e41e20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:24.689 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.696+0000 7f72eb6c8640 1 -- 172.21.15.86:0/3929770378 shutdown_connections 2024-03-29T11:30:24.689 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.696+0000 7f72eb6c8640 1 -- 172.21.15.86:0/3929770378 >> 172.21.15.86:0/3929770378 conn(0x7f72e406b660 msgr2=0x7f72e413ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:24.689 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.696+0000 7f72eb6c8640 1 -- 172.21.15.86:0/3929770378 shutdown_connections 2024-03-29T11:30:24.689 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.696+0000 7f72eb6c8640 1 -- 172.21.15.86:0/3929770378 wait complete. 2024-03-29T11:30:24.689 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:24.696+0000 7f72eb6c8640 1 librados: shutdown 2024-03-29T11:30:24.704 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:31.512 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:31.656 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.663+0000 7fca94af4640 10 monclient: get_monmap_and_config 2024-03-29T11:30:31.657 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.663+0000 7fca94af4640 10 monclient: build_initial_monmap 2024-03-29T11:30:31.657 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.663+0000 7fca94af4640 10 monclient: monmap: 2024-03-29T11:30:31.657 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:30:31.657 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:31.657 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:30:31.664676+0000 2024-03-29T11:30:31.657 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:30:31.664676+0000 2024-03-29T11:30:31.657 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:31.657 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:31.657 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:30:31.657 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:30:31.657 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:30:31.657 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:31.658 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.664+0000 7fca94af4640 1 Processor -- start 2024-03-29T11:30:31.658 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.664+0000 7fca94af4640 1 -- start start 2024-03-29T11:30:31.658 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.664+0000 7fca94af4640 10 monclient: init 2024-03-29T11:30:31.658 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.665+0000 7fca94af4640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:31.659 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.665+0000 7fca94af4640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:30:31.660 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.667+0000 7fca94af4640 10 monclient(hunting): picked mon.noname-c con 0x7fca900b0770 addr v1:172.21.15.196:6789/0 2024-03-29T11:30:31.660 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.667+0000 7fca94af4640 10 monclient(hunting): picked mon.noname-b con 0x7fca9014a700 addr v1:172.21.15.86:6790/0 2024-03-29T11:30:31.660 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.667+0000 7fca94af4640 10 monclient(hunting): picked mon.noname-a con 0x7fca900ac5a0 addr v1:172.21.15.86:6789/0 2024-03-29T11:30:31.660 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.667+0000 7fca94af4640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca900aba70 con 0x7fca900ac5a0 2024-03-29T11:30:31.660 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.667+0000 7fca94af4640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca900aecb0 con 0x7fca900b0770 2024-03-29T11:30:31.660 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.667+0000 7fca94af4640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca900af150 con 0x7fca9014a700 2024-03-29T11:30:31.660 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.667+0000 7fca94af4640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:31.660 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.667+0000 7fca94af4640 10 monclient(hunting): authenticate will time out at 1455.985200s 2024-03-29T11:30:31.661 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.668+0000 7fca8dd9b640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7fca9014a700 0x7fca900abe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:52506/0 (socket says 172.21.15.86:52506) 2024-03-29T11:30:31.661 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.668+0000 7fca8dd9b640 1 -- 172.21.15.86:0/921566983 learned_addr learned my addr 172.21.15.86:0/921566983 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:30:31.662 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.669+0000 7fca8d59a640 1 -- 172.21.15.86:0/921566983 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1040188755 0 0) 0x7fca900af150 con 0x7fca9014a700 2024-03-29T11:30:31.662 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.669+0000 7fca8d59a640 10 monclient(hunting): _negotiate 2024-03-29T11:30:31.662 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.669+0000 7fca8d59a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:31.662 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.669+0000 7fca8d59a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:31.662 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.669+0000 7fca8d59a640 10 monclient(hunting): my global_id is 4652 2024-03-29T11:30:31.662 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.669+0000 7fca8d59a640 1 -- 172.21.15.86:0/921566983 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca78003520 con 0x7fca9014a700 2024-03-29T11:30:31.664 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.669+0000 7fca8d59a640 1 -- 172.21.15.86:0/921566983 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 963024470 0 0) 0x7fca78003520 con 0x7fca9014a700 2024-03-29T11:30:31.664 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.669+0000 7fca8d59a640 1 -- 172.21.15.86:0/921566983 >> v1:172.21.15.196:6789/0 conn(0x7fca900b0770 legacy=0x7fca90149ff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:30:31.664 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.669+0000 7fca8d59a640 1 -- 172.21.15.86:0/921566983 >> v1:172.21.15.86:6789/0 conn(0x7fca900ac5a0 legacy=0x7fca90151f00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:30:31.664 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.670+0000 7fca8d59a640 10 monclient: _finish_hunting 0 2024-03-29T11:30:31.664 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.670+0000 7fca8d59a640 1 monclient: found mon.noname-b 2024-03-29T11:30:31.664 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.670+0000 7fca8d59a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:31.664 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.670+0000 7fca8d59a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:30:31.665 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.670+0000 7fca8d59a640 1 -- 172.21.15.86:0/921566983 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca900ac9d0 con 0x7fca9014a700 2024-03-29T11:30:31.665 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.670+0000 7fca8d59a640 10 monclient: _finish_auth 0 2024-03-29T11:30:31.665 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.670+0000 7fca8d59a640 10 monclient: _check_auth_tickets 2024-03-29T11:30:31.665 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.670+0000 7fca8d59a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:31.665 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.670+0000 7fca8d59a640 1 -- 172.21.15.86:0/921566983 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fca84001690 con 0x7fca9014a700 2024-03-29T11:30:31.665 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.670+0000 7fca8d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:31.665 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.670+0000 7fca8d59a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:31.665 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.670+0000 7fca8d59a640 10 monclient: dump: 2024-03-29T11:30:31.665 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:30:31.665 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:31.665 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:31.665 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:31.665 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:31.665 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:31.665 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:30:31.666 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:30:31.666 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:30:31.666 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:31.666 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.670+0000 7fca94af4640 5 monclient: authenticate success, global_id 4652 2024-03-29T11:30:31.666 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.670+0000 7fca94af4640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:31.666 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.673+0000 7fca8d59a640 1 -- 172.21.15.86:0/921566983 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca84001920 con 0x7fca9014a700 2024-03-29T11:30:31.666 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.673+0000 7fca8d59a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:31.667 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.673+0000 7fca8d59a640 1 -- 172.21.15.86:0/921566983 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fca84001cb0 con 0x7fca9014a700 2024-03-29T11:30:31.667 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.673+0000 7fca8d59a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:31.667 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.673+0000 7fca8d59a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:31.667 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.673+0000 7fca8d59a640 10 monclient: dump: 2024-03-29T11:30:31.667 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:30:31.667 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:31.667 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:31.667 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:31.667 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:31.667 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:31.667 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:30:31.667 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:30:31.667 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:30:31.667 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:31.667 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.673+0000 7fca94af4640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:31.667 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.673+0000 7fca94af4640 4 set_mon_vals no callback set 2024-03-29T11:30:31.668 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.674+0000 7fca94af4640 10 monclient: shutdown 2024-03-29T11:30:31.669 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.674+0000 7fca94af4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:31.669 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.674+0000 7fca94af4640 1 -- 172.21.15.86:0/921566983 >> v1:172.21.15.86:6790/0 conn(0x7fca9014a700 legacy=0x7fca900abe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:31.669 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.675+0000 7fca94af4640 1 -- 172.21.15.86:0/921566983 shutdown_connections 2024-03-29T11:30:31.669 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.675+0000 7fca94af4640 1 -- 172.21.15.86:0/921566983 >> 172.21.15.86:0/921566983 conn(0x7fca9006b660 msgr2=0x7fca9006ba70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:31.669 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.677+0000 7fca94af4640 1 -- 172.21.15.86:0/921566983 shutdown_connections 2024-03-29T11:30:31.671 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.678+0000 7fca94af4640 1 -- 172.21.15.86:0/921566983 wait complete. 2024-03-29T11:30:31.672 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.679+0000 7fca94af4640 10 monclient: build_initial_monmap 2024-03-29T11:30:31.673 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.679+0000 7fca94af4640 10 monclient: monmap: 2024-03-29T11:30:31.673 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:30:31.673 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:31.673 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:30:31.673 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:30:31.673 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:31.673 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:31.673 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:30:31.673 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:30:31.673 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:30:31.673 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:31.673 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.679+0000 7fca94af4640 1 librados: starting msgr at 2024-03-29T11:30:31.673 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.679+0000 7fca94af4640 1 librados: starting objecter 2024-03-29T11:30:31.673 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.679+0000 7fca94af4640 1 Processor -- start 2024-03-29T11:30:31.675 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.680+0000 7fca94af4640 1 -- start start 2024-03-29T11:30:31.675 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.680+0000 7fca94af4640 1 librados: setting wanted keys 2024-03-29T11:30:31.675 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.680+0000 7fca94af4640 1 librados: calling monclient init 2024-03-29T11:30:31.675 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.680+0000 7fca94af4640 10 monclient: init 2024-03-29T11:30:31.675 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.681+0000 7fca94af4640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:31.675 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.681+0000 7fca94af4640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:30:31.675 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.681+0000 7fca8e59c640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7fca900ac5a0 0x7fca90126990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:52928/0 (socket says 172.21.15.86:52928) 2024-03-29T11:30:31.675 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.681+0000 7fca8e59c640 1 -- 172.21.15.86:0/2060169818 learned_addr learned my addr 172.21.15.86:0/2060169818 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:30:31.675 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.681+0000 7fca7effd640 10 client.?.objecter ms_handle_connect 0x7fca900ac5a0 2024-03-29T11:30:31.675 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.681+0000 7fca7effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:31.675 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.681+0000 7fca94af4640 10 monclient(hunting): picked mon.noname-c con 0x7fca900ac5a0 addr v1:172.21.15.86:6790/0 2024-03-29T11:30:31.677 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.684+0000 7fca94af4640 10 monclient(hunting): picked mon.noname-b con 0x7fca900b0770 addr v1:172.21.15.196:6789/0 2024-03-29T11:30:31.677 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.684+0000 7fca94af4640 10 monclient(hunting): picked mon.noname-a con 0x7fca9014a700 addr v1:172.21.15.86:6789/0 2024-03-29T11:30:31.677 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.684+0000 7fca94af4640 1 -- 172.21.15.86:0/2060169818 --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca901270a0 con 0x7fca9014a700 2024-03-29T11:30:31.677 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.684+0000 7fca94af4640 1 -- 172.21.15.86:0/2060169818 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca90145960 con 0x7fca900b0770 2024-03-29T11:30:31.677 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.684+0000 7fca94af4640 1 -- 172.21.15.86:0/2060169818 --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca901239d0 con 0x7fca900ac5a0 2024-03-29T11:30:31.677 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.684+0000 7fca94af4640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:31.677 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.684+0000 7fca94af4640 10 monclient(hunting): authenticate will time out at 1456.002157s 2024-03-29T11:30:31.678 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.684+0000 7fca7effd640 10 client.?.objecter ms_handle_connect 0x7fca900b0770 2024-03-29T11:30:31.678 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.684+0000 7fca7effd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:31.680 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.685+0000 7fca7effd640 1 -- 172.21.15.86:0/2060169818 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3807110947 0 0) 0x7fca901239d0 con 0x7fca900ac5a0 2024-03-29T11:30:31.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.685+0000 7fca7effd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:31.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.685+0000 7fca7effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:31.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.685+0000 7fca7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:31.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.685+0000 7fca7effd640 10 monclient(hunting): my global_id is 4655 2024-03-29T11:30:31.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.685+0000 7fca7effd640 1 -- 172.21.15.86:0/2060169818 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca70004eb0 con 0x7fca900ac5a0 2024-03-29T11:30:31.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.685+0000 7fca7effd640 1 -- 172.21.15.86:0/2060169818 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3063956952 0 0) 0x7fca90145960 con 0x7fca900b0770 2024-03-29T11:30:31.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.685+0000 7fca7effd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:31.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.685+0000 7fca7effd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:31.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.685+0000 7fca7effd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:31.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.685+0000 7fca7effd640 10 monclient(hunting): my global_id is 4525 2024-03-29T11:30:31.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.685+0000 7fca7effd640 1 -- 172.21.15.86:0/2060169818 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fca901239d0 con 0x7fca900b0770 2024-03-29T11:30:31.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.685+0000 7fca7effd640 1 -- 172.21.15.86:0/2060169818 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1163935212 0 0) 0x7fca901239d0 con 0x7fca900b0770 2024-03-29T11:30:31.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.685+0000 7fca7effd640 1 -- 172.21.15.86:0/2060169818 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca90145960 con 0x7fca900b0770 2024-03-29T11:30:31.682 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.685+0000 7fca7effd640 1 -- 172.21.15.86:0/2060169818 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fca84001610 con 0x7fca900b0770 2024-03-29T11:30:31.682 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.685+0000 7fca7effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:31.682 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.685+0000 7fca7effd640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:31.682 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.685+0000 7fca7effd640 10 monclient(hunting): dump: 2024-03-29T11:30:31.682 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:30:31.682 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:31.682 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:31.682 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:31.682 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:31.682 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:31.682 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:30:31.682 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:30:31.682 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:30:31.682 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:31.683 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.685+0000 7fca7effd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:31.683 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.685+0000 7fca7effd640 1 -- 172.21.15.86:0/2060169818 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3264787813 0 0) 0x7fca70004eb0 con 0x7fca900ac5a0 2024-03-29T11:30:31.683 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.685+0000 7fca7effd640 1 -- 172.21.15.86:0/2060169818 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fca901239d0 con 0x7fca900ac5a0 2024-03-29T11:30:31.683 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.685+0000 7fca7effd640 1 -- 172.21.15.86:0/2060169818 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fca80001870 con 0x7fca900ac5a0 2024-03-29T11:30:31.683 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.685+0000 7fca7effd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:31.683 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.685+0000 7fca7effd640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:31.683 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.685+0000 7fca7effd640 10 monclient(hunting): dump: 2024-03-29T11:30:31.683 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:30:31.683 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:31.683 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:31.683 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:31.683 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:31.683 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:31.683 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:30:31.683 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:30:31.683 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:30:31.684 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:31.684 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.686+0000 7fca7effd640 1 -- 172.21.15.86:0/2060169818 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 554602068 0 0) 0x7fca90145960 con 0x7fca900b0770 2024-03-29T11:30:31.684 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.686+0000 7fca7effd640 1 -- 172.21.15.86:0/2060169818 >> v1:172.21.15.86:6790/0 conn(0x7fca900ac5a0 legacy=0x7fca90126990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:31.684 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.686+0000 7fca7effd640 1 -- 172.21.15.86:0/2060169818 >> v1:172.21.15.86:6789/0 conn(0x7fca9014a700 legacy=0x7fca901e5ec0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:30:31.684 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.687+0000 7fca7effd640 10 monclient: _finish_hunting 0 2024-03-29T11:30:31.684 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.687+0000 7fca7effd640 1 monclient: found mon.b 2024-03-29T11:30:31.684 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.687+0000 7fca7effd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:31.684 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.687+0000 7fca7effd640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:30:31.684 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.687+0000 7fca7effd640 1 -- 172.21.15.86:0/2060169818 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca901df550 con 0x7fca900b0770 2024-03-29T11:30:31.684 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.687+0000 7fca7effd640 10 monclient: _finish_auth 0 2024-03-29T11:30:31.684 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.687+0000 7fca7effd640 10 monclient: _check_auth_tickets 2024-03-29T11:30:31.684 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.687+0000 7fca7effd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:31.684 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.687+0000 7fca94af4640 5 monclient: authenticate success, global_id 4525 2024-03-29T11:30:31.684 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.687+0000 7fca94af4640 10 monclient: _renew_subs 2024-03-29T11:30:31.685 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.687+0000 7fca94af4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:30:31.685 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.687+0000 7fca94af4640 1 -- 172.21.15.86:0/2060169818 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca901df800 con 0x7fca900b0770 2024-03-29T11:30:31.685 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.687+0000 7fca94af4640 10 client.4525.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:31.685 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.687+0000 7fca94af4640 10 monclient: _renew_subs 2024-03-29T11:30:31.685 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.687+0000 7fca94af4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:30:31.685 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.687+0000 7fca94af4640 1 -- 172.21.15.86:0/2060169818 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fca90126110 con 0x7fca900b0770 2024-03-29T11:30:31.685 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.687+0000 7fca94af4640 1 librados: init done 2024-03-29T11:30:31.686 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.689+0000 7fca94af4640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:31.686 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.689+0000 7fca94af4640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:31.686 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.689+0000 7fca94af4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:30:31.686 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.689+0000 7fca94af4640 1 -- 172.21.15.86:0/2060169818 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca90075f10 con 0x7fca900b0770 2024-03-29T11:30:31.686 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.690+0000 7fca7effd640 1 -- 172.21.15.86:0/2060169818 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fca84004a90 con 0x7fca900b0770 2024-03-29T11:30:31.687 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.690+0000 7fca7effd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:31.687 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.690+0000 7fca7effd640 1 -- 172.21.15.86:0/2060169818 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fca84004da0 con 0x7fca900b0770 2024-03-29T11:30:31.687 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.690+0000 7fca7effd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:31.687 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.690+0000 7fca7effd640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:30:31.687 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.690+0000 7fca7effd640 10 monclient: dump: 2024-03-29T11:30:31.687 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:30:31.687 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:31.687 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:31.687 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:31.687 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:31.687 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:31.687 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:30:31.687 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:30:31.687 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:30:31.687 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:31.687 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.690+0000 7fca8cd99640 4 set_mon_vals no callback set 2024-03-29T11:30:31.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.691+0000 7fca7effd640 1 -- 172.21.15.86:0/2060169818 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7fca8401ad70 con 0x7fca900b0770 2024-03-29T11:30:31.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.691+0000 7fca7effd640 1 -- 172.21.15.86:0/2060169818 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(10..10 src has 1..10) v4 ==== 1645+0+0 (unknown 1233521 0 0) 0x7fca84084aa0 con 0x7fca900b0770 2024-03-29T11:30:31.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.691+0000 7fca7effd640 10 client.4525.objecter ms_dispatch 0x7fca90000b90 osd_map(10..10 src has 1..10) v4 2024-03-29T11:30:31.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.691+0000 7fca7effd640 3 client.4525.objecter handle_osd_map got epochs [10,10] > 0 2024-03-29T11:30:31.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.691+0000 7fca7effd640 3 client.4525.objecter handle_osd_map decoding full epoch 10 2024-03-29T11:30:31.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.692+0000 7fca7effd640 20 client.4525.objecter dump_active .. 0 homeless 2024-03-29T11:30:31.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.692+0000 7fca8e59c640 10 monclient: get_auth_request con 0x7fca7006de30 auth_method 0 2024-03-29T11:30:31.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.694+0000 7fca7effd640 10 client.4525.objecter ms_handle_connect 0x7fca7006de30 2024-03-29T11:30:31.692 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.699+0000 7fca7effd640 1 -- 172.21.15.86:0/2060169818 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fca84050380 con 0x7fca900b0770 2024-03-29T11:30:31.692 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.699+0000 7fca7effd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:31.692 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:31.699+0000 7fca7effd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:32.009 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:32.016+0000 7fca94af4640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:32.009 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:32.016+0000 7fca94af4640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:32.009 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:32.016+0000 7fca94af4640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:30:32.009 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:32.016+0000 7fca94af4640 1 -- 172.21.15.86:0/2060169818 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fca90126410 con 0x7fca900b0770 2024-03-29T11:30:32.010 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:32.017+0000 7fca7effd640 1 -- 172.21.15.86:0/2060169818 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) v1 ==== 74+0+7058 (unknown 4040575075 0 357110730) 0x7fca840522b0 con 0x7fca900b0770 2024-03-29T11:30:32.010 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:32.017+0000 7fca7effd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:32.010 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:32.017+0000 7fca7effd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:32.010 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:30:32.010 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":10,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:20:35.749273+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:20:35.749273+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:32.012 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:32.019+0000 7fca94af4640 10 librados: watch_flush enter 2024-03-29T11:30:32.013 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:32.020+0000 7fca94af4640 10 librados: watch_flush exit 2024-03-29T11:30:32.013 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:32.020+0000 7fca94af4640 20 client.4525.objecter shutdown clearing up homeless session... 2024-03-29T11:30:32.013 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:32.020+0000 7fca94af4640 10 client.4525.objecter successfully canceled tick 2024-03-29T11:30:32.013 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:32.020+0000 7fca94af4640 1 -- 172.21.15.86:0/2060169818 >> v1:172.21.15.196:6816/588110318 conn(0x7fca7006de30 legacy=0x7fca700702f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:32.013 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:32.020+0000 7fca94af4640 10 monclient: shutdown 2024-03-29T11:30:32.013 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:32.020+0000 7fca94af4640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:32.013 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:32.020+0000 7fca94af4640 1 -- 172.21.15.86:0/2060169818 >> v1:172.21.15.196:6789/0 conn(0x7fca900b0770 legacy=0x7fca901dee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:32.016 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:32.023+0000 7fca94af4640 1 -- 172.21.15.86:0/2060169818 shutdown_connections 2024-03-29T11:30:32.016 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:32.023+0000 7fca94af4640 1 -- 172.21.15.86:0/2060169818 >> 172.21.15.86:0/2060169818 conn(0x7fca9006b660 msgr2=0x7fca900b2d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:32.016 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:32.023+0000 7fca94af4640 1 -- 172.21.15.86:0/2060169818 shutdown_connections 2024-03-29T11:30:32.016 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:32.023+0000 7fca94af4640 1 -- 172.21.15.86:0/2060169818 wait complete. 2024-03-29T11:30:32.016 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:32.023+0000 7fca94af4640 1 librados: shutdown 2024-03-29T11:30:32.030 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:38.837 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:38.986 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.992+0000 7f3afbc45640 10 monclient: get_monmap_and_config 2024-03-29T11:30:38.987 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.992+0000 7f3afbc45640 10 monclient: build_initial_monmap 2024-03-29T11:30:38.987 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.992+0000 7f3afbc45640 10 monclient: monmap: 2024-03-29T11:30:38.987 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:30:38.987 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:38.987 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:30:38.994660+0000 2024-03-29T11:30:38.987 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:30:38.994660+0000 2024-03-29T11:30:38.987 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:38.987 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:38.987 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:30:38.987 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:30:38.987 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:30:38.987 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:38.988 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.993+0000 7f3afbc45640 1 Processor -- start 2024-03-29T11:30:38.988 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.993+0000 7f3afbc45640 1 -- start start 2024-03-29T11:30:38.988 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.993+0000 7f3afbc45640 10 monclient: init 2024-03-29T11:30:38.988 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.994+0000 7f3afbc45640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:38.988 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.994+0000 7f3afbc45640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:30:38.988 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.994+0000 7f3afbc45640 10 monclient(hunting): picked mon.noname-c con 0x7f3af41457f0 addr v1:172.21.15.196:6789/0 2024-03-29T11:30:38.989 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.994+0000 7f3afbc45640 10 monclient(hunting): picked mon.noname-a con 0x7f3af413f650 addr v1:172.21.15.86:6789/0 2024-03-29T11:30:38.989 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.994+0000 7f3afbc45640 10 monclient(hunting): picked mon.noname-b con 0x7f3af40a9d70 addr v1:172.21.15.86:6790/0 2024-03-29T11:30:38.989 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.994+0000 7f3afbc45640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af40aaef0 con 0x7f3af413f650 2024-03-29T11:30:38.989 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.994+0000 7f3afbc45640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af4145bd0 con 0x7f3af41457f0 2024-03-29T11:30:38.989 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.994+0000 7f3afbc45640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af4145eb0 con 0x7f3af40a9d70 2024-03-29T11:30:38.989 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.994+0000 7f3afbc45640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:38.989 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.994+0000 7f3afbc45640 10 monclient(hunting): authenticate will time out at 1463.313634s 2024-03-29T11:30:38.993 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.998+0000 7f3af99e1640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f3af41457f0 0x7f3af413ef40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:43223/0 (socket says 172.21.15.86:43223) 2024-03-29T11:30:38.993 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.998+0000 7f3af99e1640 1 -- 172.21.15.86:0/487715237 learned_addr learned my addr 172.21.15.86:0/487715237 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:30:38.993 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.999+0000 7f3af89df640 1 -- 172.21.15.86:0/487715237 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3557328752 0 0) 0x7f3af4145bd0 con 0x7f3af41457f0 2024-03-29T11:30:38.993 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.999+0000 7f3af89df640 10 monclient(hunting): _negotiate 2024-03-29T11:30:38.993 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.999+0000 7f3af89df640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:38.993 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.999+0000 7f3af89df640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:38.993 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.999+0000 7f3af89df640 10 monclient(hunting): my global_id is 4528 2024-03-29T11:30:38.993 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.999+0000 7f3af89df640 1 -- 172.21.15.86:0/487715237 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ae0003520 con 0x7f3af41457f0 2024-03-29T11:30:38.993 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.999+0000 7f3af89df640 1 -- 172.21.15.86:0/487715237 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3745546357 0 0) 0x7f3af40aaef0 con 0x7f3af413f650 2024-03-29T11:30:38.993 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.999+0000 7f3af89df640 10 monclient(hunting): _negotiate 2024-03-29T11:30:38.993 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.999+0000 7f3af89df640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:38.994 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.999+0000 7f3af89df640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:38.994 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.999+0000 7f3af89df640 10 monclient(hunting): my global_id is 4551 2024-03-29T11:30:38.994 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.999+0000 7f3af89df640 1 -- 172.21.15.86:0/487715237 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3af4145bd0 con 0x7f3af413f650 2024-03-29T11:30:38.994 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.999+0000 7f3af89df640 1 -- 172.21.15.86:0/487715237 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3855810295 0 0) 0x7f3af4145eb0 con 0x7f3af40a9d70 2024-03-29T11:30:38.994 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.999+0000 7f3af89df640 10 monclient(hunting): _negotiate 2024-03-29T11:30:38.994 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.999+0000 7f3af89df640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:38.995 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.999+0000 7f3af89df640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:38.995 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.999+0000 7f3af89df640 10 monclient(hunting): my global_id is 4658 2024-03-29T11:30:38.995 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.999+0000 7f3af89df640 1 -- 172.21.15.86:0/487715237 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ae00039d0 con 0x7f3af40a9d70 2024-03-29T11:30:38.995 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.999+0000 7f3af89df640 1 -- 172.21.15.86:0/487715237 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1827352459 0 0) 0x7f3af4145bd0 con 0x7f3af413f650 2024-03-29T11:30:38.995 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.999+0000 7f3af89df640 1 -- 172.21.15.86:0/487715237 >> v1:172.21.15.86:6790/0 conn(0x7f3af40a9d70 legacy=0x7f3af414d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:38.995 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.999+0000 7f3af89df640 1 -- 172.21.15.86:0/487715237 >> v1:172.21.15.196:6789/0 conn(0x7f3af41457f0 legacy=0x7f3af413ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:38.995 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.999+0000 7f3af89df640 10 monclient: _finish_hunting 0 2024-03-29T11:30:38.995 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.999+0000 7f3af89df640 1 monclient: found mon.noname-a 2024-03-29T11:30:38.995 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.999+0000 7f3af89df640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:38.995 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.999+0000 7f3af89df640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.86:6789/0 2024-03-29T11:30:38.995 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.999+0000 7f3af89df640 1 -- 172.21.15.86:0/487715237 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af4146340 con 0x7f3af413f650 2024-03-29T11:30:38.995 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.999+0000 7f3af89df640 10 monclient: _finish_auth 0 2024-03-29T11:30:38.995 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.999+0000 7f3af89df640 10 monclient: _check_auth_tickets 2024-03-29T11:30:38.995 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.999+0000 7f3af89df640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:38.995 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.999+0000 7f3af89df640 1 -- 172.21.15.86:0/487715237 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f3af00015f0 con 0x7f3af413f650 2024-03-29T11:30:38.996 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.999+0000 7f3af89df640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:38.996 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.999+0000 7f3af89df640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:38.996 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:38.999+0000 7f3af89df640 10 monclient: dump: 2024-03-29T11:30:38.996 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:30:38.996 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:38.996 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:38.996 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:38.996 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:38.996 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:38.996 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:30:38.996 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:30:38.996 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:30:38.996 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:38.996 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.000+0000 7f3afbc45640 5 monclient: authenticate success, global_id 4551 2024-03-29T11:30:38.996 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.000+0000 7f3afbc45640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:38.997 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.001+0000 7f3af89df640 1 -- 172.21.15.86:0/487715237 <== mon.0 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3af0001750 con 0x7f3af413f650 2024-03-29T11:30:38.997 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.001+0000 7f3af89df640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:38.997 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.001+0000 7f3afbc45640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:38.998 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.001+0000 7f3afbc45640 4 set_mon_vals no callback set 2024-03-29T11:30:38.998 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.002+0000 7f3afbc45640 10 monclient: shutdown 2024-03-29T11:30:38.998 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.002+0000 7f3afbc45640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:38.998 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.002+0000 7f3af89df640 1 -- 172.21.15.86:0/487715237 <== mon.0 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f3af0001ba0 con 0x7f3af413f650 2024-03-29T11:30:38.998 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.002+0000 7f3afbc45640 1 -- 172.21.15.86:0/487715237 >> v1:172.21.15.86:6789/0 conn(0x7f3af413f650 legacy=0x7f3af40b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:38.998 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.002+0000 7f3af89df640 10 monclient: discarding stray monitor message mon_map magic: 0 v1 2024-03-29T11:30:38.998 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.003+0000 7f3afbc45640 1 -- 172.21.15.86:0/487715237 shutdown_connections 2024-03-29T11:30:38.998 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.003+0000 7f3afbc45640 1 -- 172.21.15.86:0/487715237 >> 172.21.15.86:0/487715237 conn(0x7f3af4066a00 msgr2=0x7f3af4066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:38.998 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.004+0000 7f3afbc45640 1 -- 172.21.15.86:0/487715237 shutdown_connections 2024-03-29T11:30:38.998 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.004+0000 7f3afbc45640 1 -- 172.21.15.86:0/487715237 wait complete. 2024-03-29T11:30:38.999 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.005+0000 7f3afbc45640 10 monclient: build_initial_monmap 2024-03-29T11:30:39.000 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.005+0000 7f3afbc45640 10 monclient: monmap: 2024-03-29T11:30:39.000 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:30:39.000 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:39.000 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:30:39.000 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:30:39.000 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:39.000 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:39.000 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:30:39.000 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:30:39.000 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:30:39.001 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:39.001 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.005+0000 7f3afbc45640 1 librados: starting msgr at 2024-03-29T11:30:39.001 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.005+0000 7f3afbc45640 1 librados: starting objecter 2024-03-29T11:30:39.001 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.005+0000 7f3afbc45640 1 Processor -- start 2024-03-29T11:30:39.001 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.006+0000 7f3afbc45640 1 -- start start 2024-03-29T11:30:39.001 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.006+0000 7f3afbc45640 1 librados: setting wanted keys 2024-03-29T11:30:39.001 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.006+0000 7f3afbc45640 1 librados: calling monclient init 2024-03-29T11:30:39.001 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.006+0000 7f3afbc45640 10 monclient: init 2024-03-29T11:30:39.003 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.007+0000 7f3afbc45640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:39.003 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.007+0000 7f3afbc45640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:30:39.003 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.007+0000 7f3afbc45640 10 monclient(hunting): picked mon.noname-c con 0x7f3af40a9d70 addr v1:172.21.15.86:6790/0 2024-03-29T11:30:39.003 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.007+0000 7f3afbc45640 10 monclient(hunting): picked mon.noname-b con 0x7f3af413f650 addr v1:172.21.15.196:6789/0 2024-03-29T11:30:39.003 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.007+0000 7f3afbc45640 10 monclient(hunting): picked mon.noname-a con 0x7f3af41457f0 addr v1:172.21.15.86:6789/0 2024-03-29T11:30:39.003 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.007+0000 7f3afbc45640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af40af2b0 con 0x7f3af41457f0 2024-03-29T11:30:39.003 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.007+0000 7f3afbc45640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af413b7d0 con 0x7f3af413f650 2024-03-29T11:30:39.003 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.007+0000 7f3afbc45640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3af40b0010 con 0x7f3af40a9d70 2024-03-29T11:30:39.003 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.007+0000 7f3afbc45640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:39.003 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.007+0000 7f3afbc45640 10 monclient(hunting): authenticate will time out at 1463.326644s 2024-03-29T11:30:39.003 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.007+0000 7f3af99e1640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f3af40a9d70 0x7f3af41d7370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:49942/0 (socket says 172.21.15.86:49942) 2024-03-29T11:30:39.003 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.007+0000 7f3af99e1640 1 -- 172.21.15.86:0/79900726 learned_addr learned my addr 172.21.15.86:0/79900726 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:30:39.005 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.010+0000 7f3aea7fc640 10 client.?.objecter ms_handle_connect 0x7f3af40a9d70 2024-03-29T11:30:39.005 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.010+0000 7f3aea7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:39.005 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.010+0000 7f3aea7fc640 1 -- 172.21.15.86:0/79900726 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 211667132 0 0) 0x7f3af40b0010 con 0x7f3af40a9d70 2024-03-29T11:30:39.005 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.010+0000 7f3aea7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:30:39.005 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.010+0000 7f3aea7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:39.005 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.010+0000 7f3aea7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:39.005 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.010+0000 7f3aea7fc640 10 monclient(hunting): my global_id is 4661 2024-03-29T11:30:39.005 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.010+0000 7f3aea7fc640 1 -- 172.21.15.86:0/79900726 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3ad8004eb0 con 0x7f3af40a9d70 2024-03-29T11:30:39.005 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.011+0000 7f3aea7fc640 10 client.?.objecter ms_handle_connect 0x7f3af41457f0 2024-03-29T11:30:39.005 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.011+0000 7f3aea7fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:39.006 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.011+0000 7f3aea7fc640 1 -- 172.21.15.86:0/79900726 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3462519412 0 0) 0x7f3ad8004eb0 con 0x7f3af40a9d70 2024-03-29T11:30:39.006 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.011+0000 7f3aea7fc640 1 -- 172.21.15.86:0/79900726 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f3af40b0010 con 0x7f3af40a9d70 2024-03-29T11:30:39.006 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.011+0000 7f3aea7fc640 1 -- 172.21.15.86:0/79900726 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f3ae4001830 con 0x7f3af40a9d70 2024-03-29T11:30:39.006 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.011+0000 7f3aea7fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:39.006 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.011+0000 7f3aea7fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:39.006 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.011+0000 7f3aea7fc640 10 monclient(hunting): dump: 2024-03-29T11:30:39.006 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:30:39.006 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:39.006 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:39.006 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:39.006 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:39.006 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:39.006 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:30:39.006 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:30:39.006 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:30:39.007 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:39.007 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.011+0000 7f3aea7fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:39.009 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.011+0000 7f3aea7fc640 1 -- 172.21.15.86:0/79900726 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3271520603 0 0) 0x7f3af40b0010 con 0x7f3af40a9d70 2024-03-29T11:30:39.009 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.011+0000 7f3aea7fc640 1 -- 172.21.15.86:0/79900726 >> v1:172.21.15.196:6789/0 conn(0x7f3af413f650 legacy=0x7f3af413a9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:39.009 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.011+0000 7f3aea7fc640 1 -- 172.21.15.86:0/79900726 >> v1:172.21.15.86:6789/0 conn(0x7f3af41457f0 legacy=0x7f3af413b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:39.009 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.011+0000 7f3aea7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:30:39.009 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.011+0000 7f3aea7fc640 1 monclient: found mon.c 2024-03-29T11:30:39.009 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.011+0000 7f3aea7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:39.009 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.011+0000 7f3aea7fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:30:39.009 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.011+0000 7f3aea7fc640 1 -- 172.21.15.86:0/79900726 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3af40b02f0 con 0x7f3af40a9d70 2024-03-29T11:30:39.009 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.011+0000 7f3aea7fc640 10 monclient: _finish_auth 0 2024-03-29T11:30:39.010 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.011+0000 7f3aea7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:30:39.010 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.011+0000 7f3aea7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:39.010 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.012+0000 7f3afbc45640 5 monclient: authenticate success, global_id 4661 2024-03-29T11:30:39.010 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.012+0000 7f3afbc45640 10 monclient: _renew_subs 2024-03-29T11:30:39.010 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.012+0000 7f3afbc45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:30:39.010 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.012+0000 7f3afbc45640 1 -- 172.21.15.86:0/79900726 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3af40adf00 con 0x7f3af40a9d70 2024-03-29T11:30:39.010 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.012+0000 7f3afbc45640 10 client.4661.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:39.010 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.012+0000 7f3aea7fc640 1 -- 172.21.15.86:0/79900726 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3ae4001d60 con 0x7f3af40a9d70 2024-03-29T11:30:39.010 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.012+0000 7f3aea7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:39.010 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.012+0000 7f3aea7fc640 1 -- 172.21.15.86:0/79900726 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f3ae4004a40 con 0x7f3af40a9d70 2024-03-29T11:30:39.010 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.012+0000 7f3aea7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:39.010 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.012+0000 7f3aea7fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:39.010 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.012+0000 7f3aea7fc640 10 monclient: dump: 2024-03-29T11:30:39.010 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:30:39.010 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:39.011 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:39.011 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:39.011 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:39.011 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:39.011 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:30:39.011 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:30:39.011 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:30:39.011 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:39.011 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.012+0000 7f3afbc45640 10 monclient: _renew_subs 2024-03-29T11:30:39.011 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.012+0000 7f3afbc45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:30:39.011 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.012+0000 7f3afbc45640 1 -- 172.21.15.86:0/79900726 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f3af40aead0 con 0x7f3af40a9d70 2024-03-29T11:30:39.011 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.012+0000 7f3afbc45640 1 librados: init done 2024-03-29T11:30:39.011 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.014+0000 7f3aea7fc640 1 -- 172.21.15.86:0/79900726 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f3ae4004ba0 con 0x7f3af40a9d70 2024-03-29T11:30:39.011 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.014+0000 7f3aea7fc640 1 -- 172.21.15.86:0/79900726 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2923349858 0 0) 0x7f3ae4084a80 con 0x7f3af40a9d70 2024-03-29T11:30:39.012 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.014+0000 7f3aea7fc640 10 client.4661.objecter ms_dispatch 0x7f3af4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:39.012 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.015+0000 7f3aebfff640 4 set_mon_vals no callback set 2024-03-29T11:30:39.012 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.015+0000 7f3aea7fc640 3 client.4661.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:39.012 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.015+0000 7f3aea7fc640 3 client.4661.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:39.013 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.015+0000 7f3af91e0640 10 monclient: get_auth_request con 0x7f3ad806dcc0 auth_method 0 2024-03-29T11:30:39.013 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.015+0000 7f3aea7fc640 20 client.4661.objecter dump_active .. 0 homeless 2024-03-29T11:30:39.013 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.016+0000 7f3afbc45640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:39.013 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.016+0000 7f3afbc45640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:39.013 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.016+0000 7f3afbc45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:30:39.013 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.016+0000 7f3afbc45640 1 -- 172.21.15.86:0/79900726 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3adc005080 con 0x7f3af40a9d70 2024-03-29T11:30:39.013 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.018+0000 7f3aea7fc640 10 client.4661.objecter ms_handle_connect 0x7f3ad806dcc0 2024-03-29T11:30:39.020 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.024+0000 7f3aea7fc640 1 -- 172.21.15.86:0/79900726 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f3ae4050190 con 0x7f3af40a9d70 2024-03-29T11:30:39.020 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.024+0000 7f3aea7fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:39.020 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.024+0000 7f3aea7fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:39.339 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.345+0000 7f3afbc45640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:39.339 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.345+0000 7f3afbc45640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:39.339 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.345+0000 7f3afbc45640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:30:39.339 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.345+0000 7f3afbc45640 1 -- 172.21.15.86:0/79900726 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3adc005330 con 0x7f3af40a9d70 2024-03-29T11:30:39.340 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:30:39.340 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":11,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:30:38.448721+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:30:38.448721+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:39.341 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.346+0000 7f3aea7fc640 1 -- 172.21.15.86:0/79900726 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3394516648) 0x7f3ae4052290 con 0x7f3af40a9d70 2024-03-29T11:30:39.341 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.346+0000 7f3aea7fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:39.341 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.346+0000 7f3aea7fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:39.343 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.349+0000 7f3afbc45640 10 librados: watch_flush enter 2024-03-29T11:30:39.343 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.349+0000 7f3afbc45640 10 librados: watch_flush exit 2024-03-29T11:30:39.343 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.349+0000 7f3afbc45640 20 client.4661.objecter shutdown clearing up homeless session... 2024-03-29T11:30:39.343 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.349+0000 7f3afbc45640 10 client.4661.objecter successfully canceled tick 2024-03-29T11:30:39.343 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.349+0000 7f3afbc45640 1 -- 172.21.15.86:0/79900726 >> v1:172.21.15.196:6816/588110318 conn(0x7f3ad806dcc0 legacy=0x7f3ad8070180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:39.343 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.349+0000 7f3afbc45640 10 monclient: shutdown 2024-03-29T11:30:39.343 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.349+0000 7f3afbc45640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:39.343 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.349+0000 7f3afbc45640 1 -- 172.21.15.86:0/79900726 >> v1:172.21.15.86:6790/0 conn(0x7f3af40a9d70 legacy=0x7f3af41d7370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:39.344 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.349+0000 7f3afbc45640 1 -- 172.21.15.86:0/79900726 shutdown_connections 2024-03-29T11:30:39.344 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.349+0000 7f3afbc45640 1 -- 172.21.15.86:0/79900726 >> 172.21.15.86:0/79900726 conn(0x7f3af4066a00 msgr2=0x7f3af413cad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:39.344 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.349+0000 7f3afbc45640 1 -- 172.21.15.86:0/79900726 shutdown_connections 2024-03-29T11:30:39.344 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.350+0000 7f3afbc45640 1 -- 172.21.15.86:0/79900726 wait complete. 2024-03-29T11:30:39.344 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:39.350+0000 7f3afbc45640 1 librados: shutdown 2024-03-29T11:30:39.359 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:46.166 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:46.311 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.318+0000 7f80f359e640 10 monclient: get_monmap_and_config 2024-03-29T11:30:46.311 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.318+0000 7f80f359e640 10 monclient: build_initial_monmap 2024-03-29T11:30:46.311 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.318+0000 7f80f359e640 10 monclient: monmap: 2024-03-29T11:30:46.311 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:30:46.312 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:46.312 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:30:46.319247+0000 2024-03-29T11:30:46.312 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:30:46.319247+0000 2024-03-29T11:30:46.312 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:46.312 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:46.312 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:30:46.312 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:30:46.312 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:30:46.312 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:46.312 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.319+0000 7f80f359e640 1 Processor -- start 2024-03-29T11:30:46.313 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.319+0000 7f80f359e640 1 -- start start 2024-03-29T11:30:46.313 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.319+0000 7f80f359e640 10 monclient: init 2024-03-29T11:30:46.314 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.320+0000 7f80f359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:46.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.320+0000 7f80f359e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:30:46.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.320+0000 7f80f359e640 10 monclient(hunting): picked mon.noname-c con 0x7f80ec037230 addr v1:172.21.15.196:6789/0 2024-03-29T11:30:46.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.320+0000 7f80f359e640 10 monclient(hunting): picked mon.noname-a con 0x7f80ec033d20 addr v1:172.21.15.86:6789/0 2024-03-29T11:30:46.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.320+0000 7f80f359e640 10 monclient(hunting): picked mon.noname-b con 0x7f80ec0c90e0 addr v1:172.21.15.86:6790/0 2024-03-29T11:30:46.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.320+0000 7f80f359e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80ec031380 con 0x7f80ec033d20 2024-03-29T11:30:46.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.320+0000 7f80f359e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80ec032f10 con 0x7f80ec037230 2024-03-29T11:30:46.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.320+0000 7f80f359e640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80ec0331f0 con 0x7f80ec0c90e0 2024-03-29T11:30:46.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.320+0000 7f80f359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:46.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.320+0000 7f80f359e640 10 monclient(hunting): authenticate will time out at 1470.638985s 2024-03-29T11:30:46.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.321+0000 7f80f2d9d640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f80ec0c90e0 0x7f80ec0d7790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:49382/0 (socket says 172.21.15.86:49382) 2024-03-29T11:30:46.315 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.321+0000 7f80f2d9d640 1 -- 172.21.15.86:0/567520766 learned_addr learned my addr 172.21.15.86:0/567520766 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:30:46.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.323+0000 7f80f159a640 1 -- 172.21.15.86:0/567520766 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1962391185 0 0) 0x7f80ec0331f0 con 0x7f80ec0c90e0 2024-03-29T11:30:46.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.323+0000 7f80f159a640 10 monclient(hunting): _negotiate 2024-03-29T11:30:46.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.323+0000 7f80f159a640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:46.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.323+0000 7f80f159a640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:46.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.323+0000 7f80f159a640 10 monclient(hunting): my global_id is 4664 2024-03-29T11:30:46.316 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.323+0000 7f80f159a640 1 -- 172.21.15.86:0/567520766 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80dc003520 con 0x7f80ec0c90e0 2024-03-29T11:30:46.317 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.324+0000 7f80f159a640 1 -- 172.21.15.86:0/567520766 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3990539641 0 0) 0x7f80dc003520 con 0x7f80ec0c90e0 2024-03-29T11:30:46.317 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.324+0000 7f80f159a640 1 -- 172.21.15.86:0/567520766 >> v1:172.21.15.196:6789/0 conn(0x7f80ec037230 legacy=0x7f80ec033610 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-29T11:30:46.317 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.324+0000 7f80f159a640 1 -- 172.21.15.86:0/567520766 >> v1:172.21.15.86:6789/0 conn(0x7f80ec033d20 legacy=0x7f80ec0c89d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:30:46.319 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.325+0000 7f80f159a640 10 monclient: _finish_hunting 0 2024-03-29T11:30:46.319 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.325+0000 7f80f159a640 1 monclient: found mon.noname-b 2024-03-29T11:30:46.319 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.325+0000 7f80f159a640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:46.319 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.325+0000 7f80f159a640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:30:46.319 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.325+0000 7f80f159a640 1 -- 172.21.15.86:0/567520766 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ec031810 con 0x7f80ec0c90e0 2024-03-29T11:30:46.319 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.325+0000 7f80f159a640 10 monclient: _finish_auth 0 2024-03-29T11:30:46.319 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.325+0000 7f80f159a640 10 monclient: _check_auth_tickets 2024-03-29T11:30:46.319 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.325+0000 7f80f159a640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:46.319 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.325+0000 7f80f159a640 1 -- 172.21.15.86:0/567520766 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f80f404f730 con 0x7f80ec0c90e0 2024-03-29T11:30:46.319 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.325+0000 7f80f159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:46.320 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.325+0000 7f80f159a640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:46.320 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.325+0000 7f80f159a640 10 monclient: dump: 2024-03-29T11:30:46.320 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:30:46.320 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:46.320 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:46.320 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:46.320 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:46.320 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:46.320 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:30:46.320 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:30:46.320 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:30:46.320 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:46.320 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.325+0000 7f80f159a640 1 -- 172.21.15.86:0/567520766 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80f4075f10 con 0x7f80ec0c90e0 2024-03-29T11:30:46.321 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.325+0000 7f80f159a640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:46.321 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.325+0000 7f80f159a640 1 -- 172.21.15.86:0/567520766 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f80f40758b0 con 0x7f80ec0c90e0 2024-03-29T11:30:46.321 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.325+0000 7f80f159a640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:46.321 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.325+0000 7f80f159a640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:46.321 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.325+0000 7f80f159a640 10 monclient: dump: 2024-03-29T11:30:46.321 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:30:46.321 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:46.321 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:46.321 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:46.321 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:46.321 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:46.321 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:30:46.321 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:30:46.321 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:30:46.321 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:46.325 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.326+0000 7f80f359e640 5 monclient: authenticate success, global_id 4664 2024-03-29T11:30:46.325 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.326+0000 7f80f359e640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:46.325 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.326+0000 7f80f359e640 4 set_mon_vals no callback set 2024-03-29T11:30:46.325 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.327+0000 7f80f359e640 10 monclient: shutdown 2024-03-29T11:30:46.325 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.327+0000 7f80f359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:46.325 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.327+0000 7f80f359e640 1 -- 172.21.15.86:0/567520766 >> v1:172.21.15.86:6790/0 conn(0x7f80ec0c90e0 legacy=0x7f80ec0d7790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:46.326 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.327+0000 7f80f359e640 1 -- 172.21.15.86:0/567520766 shutdown_connections 2024-03-29T11:30:46.326 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.327+0000 7f80f359e640 1 -- 172.21.15.86:0/567520766 >> 172.21.15.86:0/567520766 conn(0x7f80ec03a400 msgr2=0x7f80ec036850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:46.326 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.327+0000 7f80f359e640 1 -- 172.21.15.86:0/567520766 shutdown_connections 2024-03-29T11:30:46.326 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.327+0000 7f80f359e640 1 -- 172.21.15.86:0/567520766 wait complete. 2024-03-29T11:30:46.326 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.328+0000 7f80f359e640 10 monclient: build_initial_monmap 2024-03-29T11:30:46.326 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.328+0000 7f80f359e640 10 monclient: monmap: 2024-03-29T11:30:46.326 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:30:46.326 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:46.326 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:30:46.326 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:30:46.326 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:46.326 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:46.326 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:30:46.326 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:30:46.327 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:30:46.327 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:46.327 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.329+0000 7f80f359e640 1 librados: starting msgr at 2024-03-29T11:30:46.327 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.329+0000 7f80f359e640 1 librados: starting objecter 2024-03-29T11:30:46.327 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.329+0000 7f80f359e640 1 Processor -- start 2024-03-29T11:30:46.327 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.329+0000 7f80f359e640 1 -- start start 2024-03-29T11:30:46.327 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.329+0000 7f80f359e640 1 librados: setting wanted keys 2024-03-29T11:30:46.327 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.329+0000 7f80f359e640 1 librados: calling monclient init 2024-03-29T11:30:46.327 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.329+0000 7f80f359e640 10 monclient: init 2024-03-29T11:30:46.327 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.330+0000 7f80f359e640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:46.327 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.330+0000 7f80f359e640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:30:46.327 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.330+0000 7f80f359e640 10 monclient(hunting): picked mon.noname-b con 0x7f80ec033d20 addr v1:172.21.15.196:6789/0 2024-03-29T11:30:46.327 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.330+0000 7f80f359e640 10 monclient(hunting): picked mon.noname-a con 0x7f80ec037230 addr v1:172.21.15.86:6789/0 2024-03-29T11:30:46.327 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.330+0000 7f80f359e640 10 monclient(hunting): picked mon.noname-c con 0x7f80ec0c90e0 addr v1:172.21.15.86:6790/0 2024-03-29T11:30:46.327 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.330+0000 7f80f359e640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80ec0cc510 con 0x7f80ec037230 2024-03-29T11:30:46.328 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.330+0000 7f80f359e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80ec0cb3c0 con 0x7f80ec033d20 2024-03-29T11:30:46.328 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.330+0000 7f80f359e640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f80ec0cb550 con 0x7f80ec0c90e0 2024-03-29T11:30:46.328 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.330+0000 7f80f359e640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:46.328 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.330+0000 7f80f359e640 10 monclient(hunting): authenticate will time out at 1470.648305s 2024-03-29T11:30:46.328 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.331+0000 7f80f2d9d640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f80ec0c90e0 0x7f80ec164b60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:50578/0 (socket says 172.21.15.86:50578) 2024-03-29T11:30:46.328 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.331+0000 7f80f2d9d640 1 -- 172.21.15.86:0/2895206878 learned_addr learned my addr 172.21.15.86:0/2895206878 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:30:46.328 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.331+0000 7f80e2ffd640 10 client.?.objecter ms_handle_connect 0x7f80ec0c90e0 2024-03-29T11:30:46.328 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.331+0000 7f80e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:46.328 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.331+0000 7f80e2ffd640 1 -- 172.21.15.86:0/2895206878 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1100275866 0 0) 0x7f80ec0cb550 con 0x7f80ec0c90e0 2024-03-29T11:30:46.328 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.331+0000 7f80e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:46.328 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.331+0000 7f80e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:46.328 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.331+0000 7f80e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:46.328 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.331+0000 7f80e2ffd640 10 monclient(hunting): my global_id is 4667 2024-03-29T11:30:46.328 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.331+0000 7f80e2ffd640 1 -- 172.21.15.86:0/2895206878 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80d4004eb0 con 0x7f80ec0c90e0 2024-03-29T11:30:46.329 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.334+0000 7f80e2ffd640 1 -- 172.21.15.86:0/2895206878 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 902612176 0 0) 0x7f80d4004eb0 con 0x7f80ec0c90e0 2024-03-29T11:30:46.329 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.334+0000 7f80e2ffd640 1 -- 172.21.15.86:0/2895206878 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80ec0cb550 con 0x7f80ec0c90e0 2024-03-29T11:30:46.329 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.334+0000 7f80e2ffd640 1 -- 172.21.15.86:0/2895206878 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f80f404f400 con 0x7f80ec0c90e0 2024-03-29T11:30:46.329 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.334+0000 7f80e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:46.329 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.335+0000 7f80e2ffd640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:30:46.329 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.335+0000 7f80e2ffd640 10 monclient(hunting): dump: 2024-03-29T11:30:46.329 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:30:46.330 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:46.330 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:46.330 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:46.330 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:46.330 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:46.330 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:30:46.330 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:30:46.330 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:30:46.330 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:46.330 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.335+0000 7f80e2ffd640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:46.330 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.335+0000 7f80e2ffd640 10 client.?.objecter ms_handle_connect 0x7f80ec037230 2024-03-29T11:30:46.330 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.335+0000 7f80e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:46.330 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.336+0000 7f80e2ffd640 1 -- 172.21.15.86:0/2895206878 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3636928943 0 0) 0x7f80ec0cc510 con 0x7f80ec037230 2024-03-29T11:30:46.330 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.336+0000 7f80e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:46.331 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.336+0000 7f80e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:46.331 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.336+0000 7f80e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:46.331 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.336+0000 7f80e2ffd640 10 monclient(hunting): my global_id is 4557 2024-03-29T11:30:46.331 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.336+0000 7f80e2ffd640 1 -- 172.21.15.86:0/2895206878 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80d4004eb0 con 0x7f80ec037230 2024-03-29T11:30:46.331 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.336+0000 7f80e2ffd640 10 client.?.objecter ms_handle_connect 0x7f80ec033d20 2024-03-29T11:30:46.333 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.336+0000 7f80e2ffd640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:46.333 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.336+0000 7f80e2ffd640 1 -- 172.21.15.86:0/2895206878 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3774433998 0 0) 0x7f80ec0cb3c0 con 0x7f80ec033d20 2024-03-29T11:30:46.333 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.336+0000 7f80e2ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:30:46.333 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.336+0000 7f80e2ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:46.333 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.336+0000 7f80e2ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:46.333 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.336+0000 7f80e2ffd640 10 monclient(hunting): my global_id is 4531 2024-03-29T11:30:46.333 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.336+0000 7f80e2ffd640 1 -- 172.21.15.86:0/2895206878 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f80d4005770 con 0x7f80ec033d20 2024-03-29T11:30:46.333 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.336+0000 7f80e2ffd640 1 -- 172.21.15.86:0/2895206878 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2272165331 0 0) 0x7f80d4004eb0 con 0x7f80ec037230 2024-03-29T11:30:46.333 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.336+0000 7f80e2ffd640 1 -- 172.21.15.86:0/2895206878 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f80d4005bc0 con 0x7f80ec037230 2024-03-29T11:30:46.334 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.336+0000 7f80e2ffd640 1 -- 172.21.15.86:0/2895206878 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f80e4001510 con 0x7f80ec037230 2024-03-29T11:30:46.334 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.336+0000 7f80e2ffd640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:46.334 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.336+0000 7f80e2ffd640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:46.334 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.336+0000 7f80e2ffd640 10 monclient(hunting): dump: 2024-03-29T11:30:46.334 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:30:46.334 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:46.334 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:46.334 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:46.334 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:46.334 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:46.334 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:30:46.334 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:30:46.334 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:30:46.334 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:46.337 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.336+0000 7f80e2ffd640 1 -- 172.21.15.86:0/2895206878 <== mon.0 v1:172.21.15.86:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 288116108 0 0) 0x7f80d4005bc0 con 0x7f80ec037230 2024-03-29T11:30:46.337 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.336+0000 7f80e2ffd640 1 -- 172.21.15.86:0/2895206878 >> v1:172.21.15.86:6790/0 conn(0x7f80ec0c90e0 legacy=0x7f80ec164b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:46.337 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.336+0000 7f80e2ffd640 1 -- 172.21.15.86:0/2895206878 >> v1:172.21.15.196:6789/0 conn(0x7f80ec033d20 legacy=0x7f80ec0c3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:46.337 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.336+0000 7f80e2ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:30:46.337 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.336+0000 7f80e2ffd640 1 monclient: found mon.a 2024-03-29T11:30:46.337 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.336+0000 7f80e2ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:46.337 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.336+0000 7f80e2ffd640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:30:46.337 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.336+0000 7f80e2ffd640 1 -- 172.21.15.86:0/2895206878 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ec0cb6e0 con 0x7f80ec037230 2024-03-29T11:30:46.338 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.336+0000 7f80e2ffd640 10 monclient: _finish_auth 0 2024-03-29T11:30:46.338 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.336+0000 7f80e2ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:30:46.338 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.336+0000 7f80e2ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:46.338 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.336+0000 7f80f359e640 5 monclient: authenticate success, global_id 4557 2024-03-29T11:30:46.338 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.336+0000 7f80f359e640 10 monclient: _renew_subs 2024-03-29T11:30:46.338 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.336+0000 7f80f359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:30:46.338 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.336+0000 7f80f359e640 1 -- 172.21.15.86:0/2895206878 --> v1:172.21.15.86:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80ec0c4250 con 0x7f80ec037230 2024-03-29T11:30:46.338 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.337+0000 7f80f359e640 10 client.4557.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:46.338 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.337+0000 7f80f359e640 10 monclient: _renew_subs 2024-03-29T11:30:46.338 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.337+0000 7f80f359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:30:46.338 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.337+0000 7f80f359e640 1 -- 172.21.15.86:0/2895206878 --> v1:172.21.15.86:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f80ec1558a0 con 0x7f80ec037230 2024-03-29T11:30:46.338 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.337+0000 7f80f359e640 1 librados: init done 2024-03-29T11:30:46.338 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.337+0000 7f80e2ffd640 1 -- 172.21.15.86:0/2895206878 <== mon.0 v1:172.21.15.86:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f80e40019c0 con 0x7f80ec037230 2024-03-29T11:30:46.338 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.337+0000 7f80e2ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:46.339 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.337+0000 7f80e2ffd640 1 -- 172.21.15.86:0/2895206878 <== mon.0 v1:172.21.15.86:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f80e4001510 con 0x7f80ec037230 2024-03-29T11:30:46.339 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.337+0000 7f80e2ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:46.339 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.337+0000 7f80e2ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:46.339 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.337+0000 7f80e2ffd640 10 monclient: dump: 2024-03-29T11:30:46.339 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:30:46.339 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:46.339 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:46.339 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:46.339 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:46.339 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:46.339 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:30:46.339 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:30:46.339 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:30:46.339 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:46.339 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.337+0000 7f80f0d99640 4 set_mon_vals no callback set 2024-03-29T11:30:46.339 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.338+0000 7f80f359e640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:46.340 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.338+0000 7f80f359e640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:46.340 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.338+0000 7f80f359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:30:46.340 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.338+0000 7f80f359e640 1 -- 172.21.15.86:0/2895206878 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80d8005080 con 0x7f80ec037230 2024-03-29T11:30:46.340 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.339+0000 7f80e2ffd640 1 -- 172.21.15.86:0/2895206878 <== mon.0 v1:172.21.15.86:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f80e4001c30 con 0x7f80ec037230 2024-03-29T11:30:46.340 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.340+0000 7f80e2ffd640 1 -- 172.21.15.86:0/2895206878 <== mon.0 v1:172.21.15.86:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2923349858 0 0) 0x7f80e40530f0 con 0x7f80ec037230 2024-03-29T11:30:46.340 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.340+0000 7f80e2ffd640 10 client.4557.objecter ms_dispatch 0x7f80f4000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:46.340 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.340+0000 7f80e2ffd640 3 client.4557.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:46.340 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.340+0000 7f80e2ffd640 3 client.4557.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:46.340 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.340+0000 7f80e2ffd640 20 client.4557.objecter dump_active .. 0 homeless 2024-03-29T11:30:46.340 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.340+0000 7f80f259c640 10 monclient: get_auth_request con 0x7f80d406e280 auth_method 0 2024-03-29T11:30:46.340 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.340+0000 7f80e2ffd640 10 client.4557.objecter ms_handle_connect 0x7f80d406e280 2024-03-29T11:30:46.344 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.351+0000 7f80e2ffd640 1 -- 172.21.15.86:0/2895206878 <== mon.0 v1:172.21.15.86:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f80e404f390 con 0x7f80ec037230 2024-03-29T11:30:46.344 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.351+0000 7f80e2ffd640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:46.344 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.351+0000 7f80e2ffd640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:46.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.688+0000 7f80f359e640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:46.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.688+0000 7f80f359e640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:46.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.688+0000 7f80f359e640 10 monclient: _send_mon_message to mon.a at v1:172.21.15.86:6789/0 2024-03-29T11:30:46.681 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.688+0000 7f80f359e640 1 -- 172.21.15.86:0/2895206878 --> v1:172.21.15.86:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80d8005330 con 0x7f80ec037230 2024-03-29T11:30:46.682 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.689+0000 7f80e2ffd640 1 -- 172.21.15.86:0/2895206878 <== mon.0 v1:172.21.15.86:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3394516648) 0x7f80e404f530 con 0x7f80ec037230 2024-03-29T11:30:46.682 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.689+0000 7f80e2ffd640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:46.682 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.689+0000 7f80e2ffd640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:46.683 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:30:46.683 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":11,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:30:38.448721+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:30:38.448721+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:46.686 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.692+0000 7f80f359e640 10 librados: watch_flush enter 2024-03-29T11:30:46.686 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.692+0000 7f80f359e640 10 librados: watch_flush exit 2024-03-29T11:30:46.686 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.692+0000 7f80f359e640 20 client.4557.objecter shutdown clearing up homeless session... 2024-03-29T11:30:46.686 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.692+0000 7f80f359e640 10 client.4557.objecter successfully canceled tick 2024-03-29T11:30:46.686 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.692+0000 7f80f359e640 1 -- 172.21.15.86:0/2895206878 >> v1:172.21.15.196:6816/588110318 conn(0x7f80d406e280 legacy=0x7f80d4070740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:46.686 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.692+0000 7f80f359e640 10 monclient: shutdown 2024-03-29T11:30:46.686 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.692+0000 7f80f359e640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:46.686 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.692+0000 7f80f359e640 1 -- 172.21.15.86:0/2895206878 >> v1:172.21.15.86:6789/0 conn(0x7f80ec037230 legacy=0x7f80ec164450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:46.686 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.693+0000 7f80f359e640 1 -- 172.21.15.86:0/2895206878 shutdown_connections 2024-03-29T11:30:46.686 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.693+0000 7f80f359e640 1 -- 172.21.15.86:0/2895206878 >> 172.21.15.86:0/2895206878 conn(0x7f80ec03a400 msgr2=0x7f80ec039150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:46.687 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.693+0000 7f80f359e640 1 -- 172.21.15.86:0/2895206878 shutdown_connections 2024-03-29T11:30:46.687 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.693+0000 7f80f359e640 1 -- 172.21.15.86:0/2895206878 wait complete. 2024-03-29T11:30:46.687 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:46.693+0000 7f80f359e640 1 librados: shutdown 2024-03-29T11:30:46.703 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:30:53.511 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:30:53.667 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.674+0000 7f8d63281640 10 monclient: get_monmap_and_config 2024-03-29T11:30:53.667 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.674+0000 7f8d63281640 10 monclient: build_initial_monmap 2024-03-29T11:30:53.667 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.674+0000 7f8d63281640 10 monclient: monmap: 2024-03-29T11:30:53.667 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:30:53.668 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:53.668 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:30:53.675273+0000 2024-03-29T11:30:53.668 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:30:53.675273+0000 2024-03-29T11:30:53.668 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:53.668 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:53.668 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:30:53.668 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:30:53.668 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:30:53.668 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:53.669 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.676+0000 7f8d63281640 1 Processor -- start 2024-03-29T11:30:53.674 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.677+0000 7f8d63281640 1 -- start start 2024-03-29T11:30:53.674 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.677+0000 7f8d63281640 10 monclient: init 2024-03-29T11:30:53.674 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.678+0000 7f8d63281640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:53.674 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.678+0000 7f8d63281640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:30:53.674 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.678+0000 7f8d63281640 10 monclient(hunting): picked mon.noname-c con 0x7f8d5c1457f0 addr v1:172.21.15.196:6789/0 2024-03-29T11:30:53.674 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.678+0000 7f8d63281640 10 monclient(hunting): picked mon.noname-a con 0x7f8d5c13f650 addr v1:172.21.15.86:6789/0 2024-03-29T11:30:53.674 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.678+0000 7f8d63281640 10 monclient(hunting): picked mon.noname-b con 0x7f8d5c0a9d70 addr v1:172.21.15.86:6790/0 2024-03-29T11:30:53.674 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.678+0000 7f8d63281640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d5c0aaef0 con 0x7f8d5c13f650 2024-03-29T11:30:53.674 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.678+0000 7f8d63281640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d5c145bd0 con 0x7f8d5c1457f0 2024-03-29T11:30:53.674 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.678+0000 7f8d63281640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d5c145eb0 con 0x7f8d5c0a9d70 2024-03-29T11:30:53.674 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.678+0000 7f8d63281640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:53.674 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.678+0000 7f8d63281640 10 monclient(hunting): authenticate will time out at 1477.996707s 2024-03-29T11:30:53.676 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.683+0000 7f8d6081c640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f8d5c13f650 0x7f8d5c0b2620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:44251/0 (socket says 172.21.15.86:44251) 2024-03-29T11:30:53.676 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.683+0000 7f8d6081c640 1 -- 172.21.15.86:0/1846257725 learned_addr learned my addr 172.21.15.86:0/1846257725 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:30:53.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.690+0000 7f8d53fff640 1 -- 172.21.15.86:0/1846257725 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2685978628 0 0) 0x7f8d5c145eb0 con 0x7f8d5c0a9d70 2024-03-29T11:30:53.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.690+0000 7f8d53fff640 10 monclient(hunting): _negotiate 2024-03-29T11:30:53.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.690+0000 7f8d53fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:53.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.690+0000 7f8d53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:53.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.690+0000 7f8d53fff640 10 monclient(hunting): my global_id is 4670 2024-03-29T11:30:53.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.690+0000 7f8d53fff640 1 -- 172.21.15.86:0/1846257725 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d48003520 con 0x7f8d5c0a9d70 2024-03-29T11:30:53.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.690+0000 7f8d53fff640 1 -- 172.21.15.86:0/1846257725 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 389576567 0 0) 0x7f8d5c145bd0 con 0x7f8d5c1457f0 2024-03-29T11:30:53.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.690+0000 7f8d53fff640 10 monclient(hunting): _negotiate 2024-03-29T11:30:53.688 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.690+0000 7f8d53fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:53.689 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.691+0000 7f8d53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:53.689 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.691+0000 7f8d53fff640 10 monclient(hunting): my global_id is 4534 2024-03-29T11:30:53.689 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.691+0000 7f8d53fff640 1 -- 172.21.15.86:0/1846257725 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d5c145eb0 con 0x7f8d5c1457f0 2024-03-29T11:30:53.689 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.691+0000 7f8d53fff640 1 -- 172.21.15.86:0/1846257725 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1709951174 0 0) 0x7f8d5c0aaef0 con 0x7f8d5c13f650 2024-03-29T11:30:53.689 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.691+0000 7f8d53fff640 10 monclient(hunting): _negotiate 2024-03-29T11:30:53.689 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.691+0000 7f8d53fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:53.689 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.691+0000 7f8d53fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:53.689 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.691+0000 7f8d53fff640 10 monclient(hunting): my global_id is 4560 2024-03-29T11:30:53.689 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.691+0000 7f8d53fff640 1 -- 172.21.15.86:0/1846257725 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d5c145bd0 con 0x7f8d5c13f650 2024-03-29T11:30:53.689 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.691+0000 7f8d53fff640 1 -- 172.21.15.86:0/1846257725 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1415695528 0 0) 0x7f8d5c145bd0 con 0x7f8d5c13f650 2024-03-29T11:30:53.689 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.691+0000 7f8d53fff640 1 -- 172.21.15.86:0/1846257725 >> v1:172.21.15.86:6790/0 conn(0x7f8d5c0a9d70 legacy=0x7f8d5c14d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:53.689 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.691+0000 7f8d53fff640 1 -- 172.21.15.86:0/1846257725 >> v1:172.21.15.196:6789/0 conn(0x7f8d5c1457f0 legacy=0x7f8d5c13ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:53.690 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.691+0000 7f8d53fff640 10 monclient: _finish_hunting 0 2024-03-29T11:30:53.690 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.691+0000 7f8d53fff640 1 monclient: found mon.noname-a 2024-03-29T11:30:53.690 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.691+0000 7f8d53fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:53.690 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.691+0000 7f8d53fff640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.86:6789/0 2024-03-29T11:30:53.690 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.691+0000 7f8d53fff640 1 -- 172.21.15.86:0/1846257725 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d5c146340 con 0x7f8d5c13f650 2024-03-29T11:30:53.690 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.691+0000 7f8d53fff640 10 monclient: _finish_auth 0 2024-03-29T11:30:53.690 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.691+0000 7f8d53fff640 10 monclient: _check_auth_tickets 2024-03-29T11:30:53.690 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.691+0000 7f8d53fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:53.690 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.691+0000 7f8d63281640 5 monclient: authenticate success, global_id 4560 2024-03-29T11:30:53.690 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.691+0000 7f8d63281640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:53.690 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.692+0000 7f8d53fff640 1 -- 172.21.15.86:0/1846257725 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f8d4c001460 con 0x7f8d5c13f650 2024-03-29T11:30:53.690 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.692+0000 7f8d53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:53.691 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.692+0000 7f8d53fff640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:30:53.691 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.692+0000 7f8d53fff640 10 monclient: dump: 2024-03-29T11:30:53.691 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:30:53.691 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:53.691 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:53.691 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:53.691 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:53.691 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:53.691 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:30:53.691 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:30:53.691 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:30:53.691 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:53.692 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.692+0000 7f8d63281640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:30:53.692 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.693+0000 7f8d53fff640 1 -- 172.21.15.86:0/1846257725 <== mon.0 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d4c001640 con 0x7f8d5c13f650 2024-03-29T11:30:53.692 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.693+0000 7f8d53fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:53.692 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.693+0000 7f8d53fff640 1 -- 172.21.15.86:0/1846257725 <== mon.0 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f8d4c001460 con 0x7f8d5c13f650 2024-03-29T11:30:53.692 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.693+0000 7f8d53fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:53.692 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.693+0000 7f8d53fff640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:30:53.692 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.693+0000 7f8d53fff640 10 monclient: dump: 2024-03-29T11:30:53.692 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:30:53.692 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:53.692 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:53.692 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:53.692 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:53.692 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:53.693 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:30:53.693 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:30:53.693 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:30:53.693 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:53.693 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.693+0000 7f8d63281640 10 monclient: get_monmap_and_config success 2024-03-29T11:30:53.693 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.693+0000 7f8d63281640 4 set_mon_vals no callback set 2024-03-29T11:30:53.693 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.694+0000 7f8d63281640 10 monclient: shutdown 2024-03-29T11:30:53.693 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.694+0000 7f8d63281640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:53.693 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.694+0000 7f8d63281640 1 -- 172.21.15.86:0/1846257725 >> v1:172.21.15.86:6789/0 conn(0x7f8d5c13f650 legacy=0x7f8d5c0b2620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:53.693 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.694+0000 7f8d63281640 1 -- 172.21.15.86:0/1846257725 shutdown_connections 2024-03-29T11:30:53.693 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.694+0000 7f8d63281640 1 -- 172.21.15.86:0/1846257725 >> 172.21.15.86:0/1846257725 conn(0x7f8d5c066a00 msgr2=0x7f8d5c066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:53.693 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.694+0000 7f8d63281640 1 -- 172.21.15.86:0/1846257725 shutdown_connections 2024-03-29T11:30:53.694 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.694+0000 7f8d63281640 1 -- 172.21.15.86:0/1846257725 wait complete. 2024-03-29T11:30:53.696 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.695+0000 7f8d63281640 10 monclient: build_initial_monmap 2024-03-29T11:30:53.696 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.695+0000 7f8d63281640 10 monclient: monmap: 2024-03-29T11:30:53.696 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:30:53.697 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:30:53.697 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:30:53.697 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:30:53.697 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:30:53.697 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:53.697 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:30:53.697 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:30:53.697 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:30:53.697 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:53.697 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.696+0000 7f8d63281640 1 librados: starting msgr at 2024-03-29T11:30:53.697 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.696+0000 7f8d63281640 1 librados: starting objecter 2024-03-29T11:30:53.697 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.696+0000 7f8d63281640 1 Processor -- start 2024-03-29T11:30:53.698 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.696+0000 7f8d63281640 1 -- start start 2024-03-29T11:30:53.698 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.696+0000 7f8d63281640 1 librados: setting wanted keys 2024-03-29T11:30:53.698 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.696+0000 7f8d63281640 1 librados: calling monclient init 2024-03-29T11:30:53.698 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.696+0000 7f8d63281640 10 monclient: init 2024-03-29T11:30:53.698 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.697+0000 7f8d63281640 10 monclient: _reopen_session rank -1 2024-03-29T11:30:53.698 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.697+0000 7f8d63281640 10 monclient: _add_conns ranks=[0,2,1] 2024-03-29T11:30:53.698 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.697+0000 7f8d63281640 10 monclient(hunting): picked mon.noname-a con 0x7f8d5c0a9d70 addr v1:172.21.15.86:6789/0 2024-03-29T11:30:53.698 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.697+0000 7f8d63281640 10 monclient(hunting): picked mon.noname-c con 0x7f8d5c13f650 addr v1:172.21.15.86:6790/0 2024-03-29T11:30:53.698 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.697+0000 7f8d63281640 10 monclient(hunting): picked mon.noname-b con 0x7f8d5c1457f0 addr v1:172.21.15.196:6789/0 2024-03-29T11:30:53.698 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.697+0000 7f8d63281640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d5c13b620 con 0x7f8d5c0a9d70 2024-03-29T11:30:53.698 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.697+0000 7f8d63281640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d5c139e70 con 0x7f8d5c1457f0 2024-03-29T11:30:53.699 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.697+0000 7f8d63281640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d5c13b850 con 0x7f8d5c13f650 2024-03-29T11:30:53.699 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.697+0000 7f8d63281640 10 monclient(hunting): _renew_subs 2024-03-29T11:30:53.699 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.697+0000 7f8d63281640 10 monclient(hunting): authenticate will time out at 1478.015247s 2024-03-29T11:30:53.699 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.701+0000 7f8d6181e640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f8d5c1457f0 0x7f8d5c1eec30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:45011/0 (socket says 172.21.15.86:45011) 2024-03-29T11:30:53.699 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.701+0000 7f8d6181e640 1 -- 172.21.15.86:0/1235274573 learned_addr learned my addr 172.21.15.86:0/1235274573 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:30:53.699 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.701+0000 7f8d51ffb640 10 client.?.objecter ms_handle_connect 0x7f8d5c1457f0 2024-03-29T11:30:53.699 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.701+0000 7f8d51ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:53.699 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.701+0000 7f8d51ffb640 1 -- 172.21.15.86:0/1235274573 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1297355969 0 0) 0x7f8d5c139e70 con 0x7f8d5c1457f0 2024-03-29T11:30:53.699 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.701+0000 7f8d51ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:30:53.699 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.701+0000 7f8d51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:53.699 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.701+0000 7f8d51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:53.699 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.701+0000 7f8d51ffb640 10 monclient(hunting): my global_id is 4537 2024-03-29T11:30:53.699 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.701+0000 7f8d51ffb640 1 -- 172.21.15.86:0/1235274573 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d40004eb0 con 0x7f8d5c1457f0 2024-03-29T11:30:53.700 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.701+0000 7f8d51ffb640 10 client.?.objecter ms_handle_connect 0x7f8d5c13f650 2024-03-29T11:30:53.700 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.701+0000 7f8d51ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:30:53.700 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.702+0000 7f8d51ffb640 1 -- 172.21.15.86:0/1235274573 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 806908373 0 0) 0x7f8d5c13b850 con 0x7f8d5c13f650 2024-03-29T11:30:53.701 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.702+0000 7f8d51ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:30:53.701 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.702+0000 7f8d51ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:30:53.701 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.702+0000 7f8d51ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:30:53.702 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.702+0000 7f8d51ffb640 10 monclient(hunting): my global_id is 4673 2024-03-29T11:30:53.702 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.702+0000 7f8d51ffb640 1 -- 172.21.15.86:0/1235274573 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d5c139e70 con 0x7f8d5c13f650 2024-03-29T11:30:53.702 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.702+0000 7f8d51ffb640 1 -- 172.21.15.86:0/1235274573 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3632076271 0 0) 0x7f8d40004eb0 con 0x7f8d5c1457f0 2024-03-29T11:30:53.702 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.702+0000 7f8d51ffb640 1 -- 172.21.15.86:0/1235274573 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d5c13b850 con 0x7f8d5c1457f0 2024-03-29T11:30:53.702 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.702+0000 7f8d51ffb640 1 -- 172.21.15.86:0/1235274573 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2716064387 0 0) 0x7f8d5c139e70 con 0x7f8d5c13f650 2024-03-29T11:30:53.702 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.702+0000 7f8d51ffb640 1 -- 172.21.15.86:0/1235274573 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f8d40005850 con 0x7f8d5c13f650 2024-03-29T11:30:53.702 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.702+0000 7f8d51ffb640 1 -- 172.21.15.86:0/1235274573 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f8d54001d30 con 0x7f8d5c1457f0 2024-03-29T11:30:53.702 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.702+0000 7f8d51ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:53.702 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.702+0000 7f8d51ffb640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:30:53.702 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.702+0000 7f8d51ffb640 10 monclient(hunting): dump: 2024-03-29T11:30:53.702 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:30:53.702 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:53.703 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:53.703 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:53.703 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:53.703 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:53.703 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:30:53.703 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:30:53.703 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:30:53.703 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:53.703 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.702+0000 7f8d51ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:30:53.703 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.702+0000 7f8d51ffb640 1 -- 172.21.15.86:0/1235274573 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f8d4c001bb0 con 0x7f8d5c13f650 2024-03-29T11:30:53.703 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.702+0000 7f8d51ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:53.703 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.702+0000 7f8d51ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:53.704 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.702+0000 7f8d51ffb640 10 monclient(hunting): dump: 2024-03-29T11:30:53.704 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:30:53.704 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:53.704 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:53.704 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:53.704 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:53.704 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:53.704 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:30:53.704 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:30:53.704 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:30:53.704 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:53.704 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.703+0000 7f8d51ffb640 1 -- 172.21.15.86:0/1235274573 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 302755710 0 0) 0x7f8d40005850 con 0x7f8d5c13f650 2024-03-29T11:30:53.704 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.703+0000 7f8d51ffb640 1 -- 172.21.15.86:0/1235274573 >> v1:172.21.15.196:6789/0 conn(0x7f8d5c1457f0 legacy=0x7f8d5c1eec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:53.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.703+0000 7f8d51ffb640 1 -- 172.21.15.86:0/1235274573 >> v1:172.21.15.86:6789/0 conn(0x7f8d5c0a9d70 legacy=0x7f8d5c143a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:53.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.703+0000 7f8d51ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:30:53.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.703+0000 7f8d51ffb640 1 monclient: found mon.c 2024-03-29T11:30:53.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.703+0000 7f8d51ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:30:53.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.703+0000 7f8d51ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:30:53.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.703+0000 7f8d51ffb640 1 -- 172.21.15.86:0/1235274573 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d5c13bb30 con 0x7f8d5c13f650 2024-03-29T11:30:53.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.703+0000 7f8d51ffb640 10 monclient: _finish_auth 0 2024-03-29T11:30:53.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.703+0000 7f8d51ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:30:53.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.703+0000 7f8d51ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:30:53.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.703+0000 7f8d51ffb640 1 -- 172.21.15.86:0/1235274573 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f8d4c004a90 con 0x7f8d5c13f650 2024-03-29T11:30:53.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.703+0000 7f8d51ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:30:53.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.703+0000 7f8d537fe640 4 set_mon_vals no callback set 2024-03-29T11:30:53.705 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.704+0000 7f8d51ffb640 1 -- 172.21.15.86:0/1235274573 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f8d4c001bb0 con 0x7f8d5c13f650 2024-03-29T11:30:53.706 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.704+0000 7f8d51ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:30:53.706 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.704+0000 7f8d51ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:30:53.706 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.704+0000 7f8d51ffb640 10 monclient: dump: 2024-03-29T11:30:53.706 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:30:53.706 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:30:53.706 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:53.706 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:30:53.706 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:30:53.706 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:30:53.706 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:30:53.706 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:30:53.706 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:30:53.707 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:30:53.707 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.705+0000 7f8d63281640 5 monclient: authenticate success, global_id 4673 2024-03-29T11:30:53.707 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.705+0000 7f8d63281640 10 monclient: _renew_subs 2024-03-29T11:30:53.707 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.705+0000 7f8d63281640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:30:53.707 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.705+0000 7f8d63281640 1 -- 172.21.15.86:0/1235274573 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d5c13fda0 con 0x7f8d5c13f650 2024-03-29T11:30:53.707 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.706+0000 7f8d63281640 10 client.4673.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:30:53.707 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.706+0000 7f8d63281640 10 monclient: _renew_subs 2024-03-29T11:30:53.707 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.706+0000 7f8d63281640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:30:53.707 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.706+0000 7f8d63281640 1 -- 172.21.15.86:0/1235274573 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d5c13a830 con 0x7f8d5c13f650 2024-03-29T11:30:53.707 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.706+0000 7f8d63281640 1 librados: init done 2024-03-29T11:30:53.707 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.707+0000 7f8d63281640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:53.707 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.707+0000 7f8d63281640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:53.708 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.707+0000 7f8d63281640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:30:53.708 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.707+0000 7f8d63281640 1 -- 172.21.15.86:0/1235274573 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d5c139c00 con 0x7f8d5c13f650 2024-03-29T11:30:53.708 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.708+0000 7f8d51ffb640 1 -- 172.21.15.86:0/1235274573 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f8d4c01aaf0 con 0x7f8d5c13f650 2024-03-29T11:30:53.708 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.710+0000 7f8d51ffb640 1 -- 172.21.15.86:0/1235274573 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2923349858 0 0) 0x7f8d4c084500 con 0x7f8d5c13f650 2024-03-29T11:30:53.708 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.710+0000 7f8d51ffb640 10 client.4673.objecter ms_dispatch 0x7f8d5c000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:30:53.709 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.710+0000 7f8d51ffb640 3 client.4673.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:30:53.709 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.710+0000 7f8d51ffb640 3 client.4673.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:30:53.709 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.710+0000 7f8d51ffb640 20 client.4673.objecter dump_active .. 0 homeless 2024-03-29T11:30:53.709 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.711+0000 7f8d6101d640 10 monclient: get_auth_request con 0x7f8d4006e500 auth_method 0 2024-03-29T11:30:53.709 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.711+0000 7f8d51ffb640 10 client.4673.objecter ms_handle_connect 0x7f8d4006e500 2024-03-29T11:30:53.715 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.719+0000 7f8d51ffb640 1 -- 172.21.15.86:0/1235274573 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f8d4c04fd50 con 0x7f8d5c13f650 2024-03-29T11:30:53.715 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.719+0000 7f8d51ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:30:53.716 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:53.719+0000 7f8d51ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:30:54.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:54.057+0000 7f8d63281640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:54.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:54.057+0000 7f8d63281640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:54.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:54.057+0000 7f8d63281640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:30:54.050 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:54.057+0000 7f8d63281640 1 -- 172.21.15.86:0/1235274573 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8d5c139c00 con 0x7f8d5c13f650 2024-03-29T11:30:54.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:54.058+0000 7f8d51ffb640 1 -- 172.21.15.86:0/1235274573 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3394516648) 0x7f8d4c000fe0 con 0x7f8d5c13f650 2024-03-29T11:30:54.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:54.058+0000 7f8d51ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:30:54.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:54.058+0000 7f8d51ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:30:54.052 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:30:54.052 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":11,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:30:38.448721+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:30:38.448721+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:30:54.056 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:54.063+0000 7f8d63281640 10 librados: watch_flush enter 2024-03-29T11:30:54.056 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:54.063+0000 7f8d63281640 10 librados: watch_flush exit 2024-03-29T11:30:54.056 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:54.063+0000 7f8d63281640 20 client.4673.objecter shutdown clearing up homeless session... 2024-03-29T11:30:54.057 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:54.063+0000 7f8d63281640 10 client.4673.objecter successfully canceled tick 2024-03-29T11:30:54.057 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:54.063+0000 7f8d63281640 1 -- 172.21.15.86:0/1235274573 >> v1:172.21.15.196:6816/588110318 conn(0x7f8d4006e500 legacy=0x7f8d400709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:54.057 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:54.063+0000 7f8d63281640 10 monclient: shutdown 2024-03-29T11:30:54.057 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:54.063+0000 7f8d63281640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:30:54.057 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:54.063+0000 7f8d63281640 1 -- 172.21.15.86:0/1235274573 >> v1:172.21.15.86:6790/0 conn(0x7f8d5c13f650 legacy=0x7f8d5c144180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:30:54.057 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:54.063+0000 7f8d63281640 1 -- 172.21.15.86:0/1235274573 shutdown_connections 2024-03-29T11:30:54.057 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:54.063+0000 7f8d63281640 1 -- 172.21.15.86:0/1235274573 >> 172.21.15.86:0/1235274573 conn(0x7f8d5c066a00 msgr2=0x7f8d5c0b2520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:30:54.057 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:54.064+0000 7f8d63281640 1 -- 172.21.15.86:0/1235274573 shutdown_connections 2024-03-29T11:30:54.058 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:54.064+0000 7f8d63281640 1 -- 172.21.15.86:0/1235274573 wait complete. 2024-03-29T11:30:54.058 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:30:54.064+0000 7f8d63281640 1 librados: shutdown 2024-03-29T11:30:54.078 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:31:00.885 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:01.037 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.043+0000 7fc655b6d640 10 monclient: get_monmap_and_config 2024-03-29T11:31:01.037 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.043+0000 7fc655b6d640 10 monclient: build_initial_monmap 2024-03-29T11:31:01.037 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.043+0000 7fc655b6d640 10 monclient: monmap: 2024-03-29T11:31:01.037 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:31:01.037 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:31:01.037 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:31:01.044537+0000 2024-03-29T11:31:01.037 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:31:01.044537+0000 2024-03-29T11:31:01.037 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:01.037 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:31:01.037 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:31:01.038 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:31:01.038 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:31:01.038 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:31:01.039 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.044+0000 7fc655b6d640 1 Processor -- start 2024-03-29T11:31:01.039 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.044+0000 7fc655b6d640 1 -- start start 2024-03-29T11:31:01.039 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.044+0000 7fc655b6d640 10 monclient: init 2024-03-29T11:31:01.039 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.045+0000 7fc655b6d640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:01.039 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.045+0000 7fc655b6d640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:31:01.039 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.045+0000 7fc655b6d640 10 monclient(hunting): picked mon.noname-c con 0x7fc6501422b0 addr v1:172.21.15.196:6789/0 2024-03-29T11:31:01.039 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.045+0000 7fc655b6d640 10 monclient(hunting): picked mon.noname-b con 0x7fc650142690 addr v1:172.21.15.86:6790/0 2024-03-29T11:31:01.039 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.045+0000 7fc655b6d640 10 monclient(hunting): picked mon.noname-a con 0x7fc6500b17d0 addr v1:172.21.15.86:6789/0 2024-03-29T11:31:01.039 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.045+0000 7fc655b6d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc650142c80 con 0x7fc6500b17d0 2024-03-29T11:31:01.039 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.045+0000 7fc655b6d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6500aae40 con 0x7fc6501422b0 2024-03-29T11:31:01.039 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.045+0000 7fc655b6d640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc65013fbd0 con 0x7fc650142690 2024-03-29T11:31:01.040 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.045+0000 7fc655b6d640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:01.040 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.045+0000 7fc655b6d640 10 monclient(hunting): authenticate will time out at 1485.363668s 2024-03-29T11:31:01.040 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.045+0000 7fc64f7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fc6501422b0 0x7fc65013f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:51232/0 (socket says 172.21.15.86:51232) 2024-03-29T11:31:01.040 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.045+0000 7fc64f7fe640 1 -- 172.21.15.86:0/1456555658 learned_addr learned my addr 172.21.15.86:0/1456555658 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:31:01.041 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.046+0000 7fc64e7fc640 1 -- 172.21.15.86:0/1456555658 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1800194901 0 0) 0x7fc6500aae40 con 0x7fc6501422b0 2024-03-29T11:31:01.041 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.046+0000 7fc64e7fc640 10 monclient(hunting): _negotiate 2024-03-29T11:31:01.041 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.046+0000 7fc64e7fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:01.041 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.046+0000 7fc64e7fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:01.041 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.046+0000 7fc64e7fc640 10 monclient(hunting): my global_id is 4540 2024-03-29T11:31:01.041 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.046+0000 7fc64e7fc640 1 -- 172.21.15.86:0/1456555658 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc638003520 con 0x7fc6501422b0 2024-03-29T11:31:01.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.046+0000 7fc64e7fc640 1 -- 172.21.15.86:0/1456555658 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3353318731 0 0) 0x7fc638003520 con 0x7fc6501422b0 2024-03-29T11:31:01.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.046+0000 7fc64e7fc640 1 -- 172.21.15.86:0/1456555658 >> v1:172.21.15.86:6790/0 conn(0x7fc650142690 legacy=0x7fc6500b2590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:01.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.046+0000 7fc64e7fc640 1 -- 172.21.15.86:0/1456555658 >> v1:172.21.15.86:6789/0 conn(0x7fc6500b17d0 legacy=0x7fc6500b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:01.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.046+0000 7fc64e7fc640 10 monclient: _finish_hunting 0 2024-03-29T11:31:01.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.046+0000 7fc64e7fc640 1 monclient: found mon.noname-c 2024-03-29T11:31:01.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.046+0000 7fc64e7fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:01.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.046+0000 7fc64e7fc640 10 monclient: _send_mon_message to mon.noname-c at v1:172.21.15.196:6789/0 2024-03-29T11:31:01.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.046+0000 7fc64e7fc640 1 -- 172.21.15.86:0/1456555658 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc650151940 con 0x7fc6501422b0 2024-03-29T11:31:01.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.046+0000 7fc64e7fc640 10 monclient: _finish_auth 0 2024-03-29T11:31:01.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.046+0000 7fc64e7fc640 10 monclient: _check_auth_tickets 2024-03-29T11:31:01.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.046+0000 7fc64e7fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:01.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.046+0000 7fc655b6d640 5 monclient: authenticate success, global_id 4540 2024-03-29T11:31:01.042 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.046+0000 7fc655b6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:01.043 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.046+0000 7fc64e7fc640 1 -- 172.21.15.86:0/1456555658 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc648001690 con 0x7fc6501422b0 2024-03-29T11:31:01.043 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.046+0000 7fc64e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:01.043 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.047+0000 7fc64e7fc640 10 monclient: got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:31:01.043 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.047+0000 7fc64e7fc640 10 monclient: dump: 2024-03-29T11:31:01.043 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:31:01.043 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:31:01.043 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:01.043 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:01.043 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:01.043 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:31:01.043 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:31:01.043 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:31:01.043 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:31:01.043 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:31:01.043 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.047+0000 7fc655b6d640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:01.043 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.047+0000 7fc64e7fc640 1 -- 172.21.15.86:0/1456555658 <== mon.1 v1:172.21.15.196:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc6480018b0 con 0x7fc6501422b0 2024-03-29T11:31:01.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.047+0000 7fc64e7fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:01.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.047+0000 7fc64e7fc640 1 -- 172.21.15.86:0/1456555658 <== mon.1 v1:172.21.15.196:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc648001690 con 0x7fc6501422b0 2024-03-29T11:31:01.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.047+0000 7fc64e7fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:01.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.047+0000 7fc64e7fc640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:31:01.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.047+0000 7fc64e7fc640 10 monclient: dump: 2024-03-29T11:31:01.044 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:31:01.044 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:31:01.044 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:01.044 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:01.044 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:01.044 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:31:01.044 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:31:01.044 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:31:01.044 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:31:01.044 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:31:01.044 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.047+0000 7fc655b6d640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:01.045 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.047+0000 7fc655b6d640 4 set_mon_vals no callback set 2024-03-29T11:31:01.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.048+0000 7fc655b6d640 10 monclient: shutdown 2024-03-29T11:31:01.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.048+0000 7fc655b6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:01.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.048+0000 7fc655b6d640 1 -- 172.21.15.86:0/1456555658 >> v1:172.21.15.196:6789/0 conn(0x7fc6501422b0 legacy=0x7fc65013f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:01.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.048+0000 7fc655b6d640 1 -- 172.21.15.86:0/1456555658 shutdown_connections 2024-03-29T11:31:01.052 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.048+0000 7fc655b6d640 1 -- 172.21.15.86:0/1456555658 >> 172.21.15.86:0/1456555658 conn(0x7fc650066a00 msgr2=0x7fc650066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:01.053 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.048+0000 7fc655b6d640 1 -- 172.21.15.86:0/1456555658 shutdown_connections 2024-03-29T11:31:01.053 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.048+0000 7fc655b6d640 1 -- 172.21.15.86:0/1456555658 wait complete. 2024-03-29T11:31:01.053 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.049+0000 7fc655b6d640 10 monclient: build_initial_monmap 2024-03-29T11:31:01.053 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.049+0000 7fc655b6d640 10 monclient: monmap: 2024-03-29T11:31:01.053 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:31:01.053 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:01.053 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:31:01.053 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:31:01.053 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:01.053 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:31:01.053 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:31:01.053 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:31:01.053 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:31:01.053 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:31:01.053 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.050+0000 7fc655b6d640 1 librados: starting msgr at 2024-03-29T11:31:01.054 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.050+0000 7fc655b6d640 1 librados: starting objecter 2024-03-29T11:31:01.054 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.050+0000 7fc655b6d640 1 Processor -- start 2024-03-29T11:31:01.054 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.050+0000 7fc655b6d640 1 -- start start 2024-03-29T11:31:01.054 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.050+0000 7fc655b6d640 1 librados: setting wanted keys 2024-03-29T11:31:01.054 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.050+0000 7fc655b6d640 1 librados: calling monclient init 2024-03-29T11:31:01.054 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.050+0000 7fc655b6d640 10 monclient: init 2024-03-29T11:31:01.054 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.051+0000 7fc655b6d640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:01.054 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.051+0000 7fc655b6d640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:31:01.054 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.051+0000 7fc655b6d640 10 monclient(hunting): picked mon.noname-a con 0x7fc6500b17d0 addr v1:172.21.15.86:6789/0 2024-03-29T11:31:01.054 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.051+0000 7fc655b6d640 10 monclient(hunting): picked mon.noname-b con 0x7fc6501422b0 addr v1:172.21.15.196:6789/0 2024-03-29T11:31:01.054 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.051+0000 7fc655b6d640 10 monclient(hunting): picked mon.noname-c con 0x7fc650142690 addr v1:172.21.15.86:6790/0 2024-03-29T11:31:01.054 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.051+0000 7fc655b6d640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc65014e250 con 0x7fc6500b17d0 2024-03-29T11:31:01.054 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.051+0000 7fc655b6d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc65014de00 con 0x7fc6501422b0 2024-03-29T11:31:01.054 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.051+0000 7fc655b6d640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc65014df90 con 0x7fc650142690 2024-03-29T11:31:01.055 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.051+0000 7fc655b6d640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:01.055 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.051+0000 7fc655b6d640 10 monclient(hunting): authenticate will time out at 1485.369985s 2024-03-29T11:31:01.055 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.052+0000 7fc64effd640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fc6501422b0 0x7fc6501defb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:51652/0 (socket says 172.21.15.86:51652) 2024-03-29T11:31:01.055 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.052+0000 7fc64effd640 1 -- 172.21.15.86:0/125017478 learned_addr learned my addr 172.21.15.86:0/125017478 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:31:01.055 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.052+0000 7fc637fff640 10 client.?.objecter ms_handle_connect 0x7fc6501422b0 2024-03-29T11:31:01.055 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.052+0000 7fc637fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:01.055 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.052+0000 7fc637fff640 10 client.?.objecter ms_handle_connect 0x7fc6500b17d0 2024-03-29T11:31:01.055 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.052+0000 7fc637fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:01.055 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.052+0000 7fc637fff640 10 client.?.objecter ms_handle_connect 0x7fc650142690 2024-03-29T11:31:01.055 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.052+0000 7fc637fff640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:01.055 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.053+0000 7fc637fff640 1 -- 172.21.15.86:0/125017478 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1741052803 0 0) 0x7fc65014e250 con 0x7fc6500b17d0 2024-03-29T11:31:01.055 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.053+0000 7fc637fff640 10 monclient(hunting): _negotiate 2024-03-29T11:31:01.055 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.053+0000 7fc637fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:01.055 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.053+0000 7fc637fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:01.056 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.053+0000 7fc637fff640 10 monclient(hunting): my global_id is 4566 2024-03-29T11:31:01.056 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.053+0000 7fc637fff640 1 -- 172.21.15.86:0/125017478 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc63c004eb0 con 0x7fc6500b17d0 2024-03-29T11:31:01.056 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.053+0000 7fc637fff640 1 -- 172.21.15.86:0/125017478 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4125517549 0 0) 0x7fc65014de00 con 0x7fc6501422b0 2024-03-29T11:31:01.056 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.053+0000 7fc637fff640 10 monclient(hunting): _negotiate 2024-03-29T11:31:01.056 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.053+0000 7fc637fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:01.056 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.053+0000 7fc637fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:01.056 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.053+0000 7fc637fff640 10 monclient(hunting): my global_id is 4543 2024-03-29T11:31:01.056 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.053+0000 7fc637fff640 1 -- 172.21.15.86:0/125017478 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc65014e250 con 0x7fc6501422b0 2024-03-29T11:31:01.056 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.053+0000 7fc637fff640 1 -- 172.21.15.86:0/125017478 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1416242817 0 0) 0x7fc65014df90 con 0x7fc650142690 2024-03-29T11:31:01.056 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.053+0000 7fc637fff640 10 monclient(hunting): _negotiate 2024-03-29T11:31:01.056 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.053+0000 7fc637fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:01.056 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.053+0000 7fc637fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:01.056 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.053+0000 7fc637fff640 10 monclient(hunting): my global_id is 4676 2024-03-29T11:31:01.056 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.053+0000 7fc637fff640 1 -- 172.21.15.86:0/125017478 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc65014de00 con 0x7fc650142690 2024-03-29T11:31:01.056 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.054+0000 7fc637fff640 1 -- 172.21.15.86:0/125017478 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2853878574 0 0) 0x7fc65014e250 con 0x7fc6501422b0 2024-03-29T11:31:01.057 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.054+0000 7fc637fff640 1 -- 172.21.15.86:0/125017478 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc65014df90 con 0x7fc6501422b0 2024-03-29T11:31:01.057 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.054+0000 7fc637fff640 1 -- 172.21.15.86:0/125017478 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc644003f20 con 0x7fc6501422b0 2024-03-29T11:31:01.057 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.054+0000 7fc637fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:01.057 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.054+0000 7fc637fff640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:31:01.057 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.054+0000 7fc637fff640 10 monclient(hunting): dump: 2024-03-29T11:31:01.057 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:31:01.057 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:31:01.057 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:01.057 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:01.057 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:01.057 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:31:01.057 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:31:01.057 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:31:01.057 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:31:01.057 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:31:01.058 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.054+0000 7fc637fff640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:01.058 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.054+0000 7fc637fff640 1 -- 172.21.15.86:0/125017478 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3597928431 0 0) 0x7fc65014de00 con 0x7fc650142690 2024-03-29T11:31:01.058 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.054+0000 7fc637fff640 1 -- 172.21.15.86:0/125017478 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc65014e250 con 0x7fc650142690 2024-03-29T11:31:01.058 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.054+0000 7fc637fff640 1 -- 172.21.15.86:0/125017478 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc640001b70 con 0x7fc650142690 2024-03-29T11:31:01.058 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.054+0000 7fc637fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:01.058 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.054+0000 7fc637fff640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:01.058 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.054+0000 7fc637fff640 10 monclient(hunting): dump: 2024-03-29T11:31:01.058 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:31:01.058 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:31:01.058 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:01.058 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:01.058 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:01.058 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:31:01.058 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:31:01.059 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:31:01.059 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:31:01.059 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:31:01.059 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.054+0000 7fc637fff640 1 -- 172.21.15.86:0/125017478 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2597499541 0 0) 0x7fc63c004eb0 con 0x7fc6500b17d0 2024-03-29T11:31:01.059 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.054+0000 7fc637fff640 1 -- 172.21.15.86:0/125017478 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7fc65014de00 con 0x7fc6500b17d0 2024-03-29T11:31:01.059 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.055+0000 7fc637fff640 1 -- 172.21.15.86:0/125017478 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc648001690 con 0x7fc6500b17d0 2024-03-29T11:31:01.059 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.055+0000 7fc637fff640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:01.059 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.055+0000 7fc637fff640 10 monclient(hunting): got monmap 1 from mon.a (according to old e1) 2024-03-29T11:31:01.059 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.055+0000 7fc637fff640 10 monclient(hunting): dump: 2024-03-29T11:31:01.059 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:31:01.059 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:31:01.059 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:01.059 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:01.059 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:01.060 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:31:01.060 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:31:01.060 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:31:01.060 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:31:01.060 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:31:01.060 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.055+0000 7fc637fff640 1 -- 172.21.15.86:0/125017478 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1234531360 0 0) 0x7fc65014df90 con 0x7fc6501422b0 2024-03-29T11:31:01.060 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.055+0000 7fc637fff640 1 -- 172.21.15.86:0/125017478 >> v1:172.21.15.86:6790/0 conn(0x7fc650142690 legacy=0x7fc6501e72f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:01.060 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.055+0000 7fc637fff640 1 -- 172.21.15.86:0/125017478 >> v1:172.21.15.86:6789/0 conn(0x7fc6500b17d0 legacy=0x7fc65012ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:01.060 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.055+0000 7fc637fff640 10 monclient: _finish_hunting 0 2024-03-29T11:31:01.060 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.055+0000 7fc637fff640 1 monclient: found mon.b 2024-03-29T11:31:01.060 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.055+0000 7fc637fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:01.060 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.055+0000 7fc637fff640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:31:01.060 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.055+0000 7fc637fff640 1 -- 172.21.15.86:0/125017478 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6500b5270 con 0x7fc6501422b0 2024-03-29T11:31:01.060 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.055+0000 7fc637fff640 10 monclient: _finish_auth 0 2024-03-29T11:31:01.060 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.055+0000 7fc637fff640 10 monclient: _check_auth_tickets 2024-03-29T11:31:01.060 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.055+0000 7fc637fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:01.061 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.055+0000 7fc655b6d640 5 monclient: authenticate success, global_id 4543 2024-03-29T11:31:01.061 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.055+0000 7fc655b6d640 10 monclient: _renew_subs 2024-03-29T11:31:01.061 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.055+0000 7fc655b6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:31:01.061 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.055+0000 7fc655b6d640 1 -- 172.21.15.86:0/125017478 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc65014d360 con 0x7fc6501422b0 2024-03-29T11:31:01.061 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.055+0000 7fc655b6d640 10 client.4543.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:01.061 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.055+0000 7fc655b6d640 10 monclient: _renew_subs 2024-03-29T11:31:01.061 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.055+0000 7fc655b6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:31:01.061 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.055+0000 7fc655b6d640 1 -- 172.21.15.86:0/125017478 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7fc65012f5a0 con 0x7fc6501422b0 2024-03-29T11:31:01.061 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.055+0000 7fc655b6d640 1 librados: init done 2024-03-29T11:31:01.061 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.055+0000 7fc637fff640 1 -- 172.21.15.86:0/125017478 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7fc644004430 con 0x7fc6501422b0 2024-03-29T11:31:01.061 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.055+0000 7fc637fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:01.061 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.055+0000 7fc64dffb640 4 set_mon_vals no callback set 2024-03-29T11:31:01.061 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.056+0000 7fc637fff640 1 -- 172.21.15.86:0/125017478 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7fc644001070 con 0x7fc6501422b0 2024-03-29T11:31:01.061 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.056+0000 7fc637fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:01.061 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.056+0000 7fc637fff640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:31:01.062 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.056+0000 7fc637fff640 10 monclient: dump: 2024-03-29T11:31:01.062 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:31:01.062 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:31:01.062 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:01.062 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:01.062 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:01.062 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:31:01.062 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:31:01.062 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:31:01.062 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:31:01.062 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:31:01.062 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.057+0000 7fc655b6d640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:01.062 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.057+0000 7fc655b6d640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:01.062 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.057+0000 7fc655b6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:31:01.062 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.057+0000 7fc655b6d640 1 -- 172.21.15.86:0/125017478 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc630005080 con 0x7fc6501422b0 2024-03-29T11:31:01.063 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.058+0000 7fc637fff640 1 -- 172.21.15.86:0/125017478 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7fc6440011d0 con 0x7fc6501422b0 2024-03-29T11:31:01.063 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.058+0000 7fc637fff640 1 -- 172.21.15.86:0/125017478 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2923349858 0 0) 0x7fc6440848b0 con 0x7fc6501422b0 2024-03-29T11:31:01.063 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.058+0000 7fc637fff640 10 client.4543.objecter ms_dispatch 0x7fc650000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:31:01.063 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.058+0000 7fc637fff640 3 client.4543.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:31:01.063 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.058+0000 7fc637fff640 3 client.4543.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:31:01.063 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.058+0000 7fc637fff640 20 client.4543.objecter dump_active .. 0 homeless 2024-03-29T11:31:01.063 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.058+0000 7fc64f7fe640 10 monclient: get_auth_request con 0x7fc63c06e0c0 auth_method 0 2024-03-29T11:31:01.063 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.059+0000 7fc637fff640 10 client.4543.objecter ms_handle_connect 0x7fc63c06e0c0 2024-03-29T11:31:01.063 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.064+0000 7fc637fff640 1 -- 172.21.15.86:0/125017478 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7fc644050190 con 0x7fc6501422b0 2024-03-29T11:31:01.064 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.064+0000 7fc637fff640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:01.064 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.064+0000 7fc637fff640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:01.455 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.461+0000 7fc655b6d640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:01.455 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.461+0000 7fc655b6d640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:01.455 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.461+0000 7fc655b6d640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:31:01.455 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.461+0000 7fc655b6d640 1 -- 172.21.15.86:0/125017478 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc630005330 con 0x7fc6501422b0 2024-03-29T11:31:01.456 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.463+0000 7fc637fff640 1 -- 172.21.15.86:0/125017478 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3394516648) 0x7fc6440520c0 con 0x7fc6501422b0 2024-03-29T11:31:01.456 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.463+0000 7fc637fff640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:01.456 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.463+0000 7fc637fff640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:01.457 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:31:01.457 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":11,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:30:38.448721+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:30:38.448721+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:01.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.467+0000 7fc655b6d640 10 librados: watch_flush enter 2024-03-29T11:31:01.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.467+0000 7fc655b6d640 10 librados: watch_flush exit 2024-03-29T11:31:01.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.467+0000 7fc655b6d640 20 client.4543.objecter shutdown clearing up homeless session... 2024-03-29T11:31:01.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.467+0000 7fc655b6d640 10 client.4543.objecter successfully canceled tick 2024-03-29T11:31:01.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.467+0000 7fc655b6d640 1 -- 172.21.15.86:0/125017478 >> v1:172.21.15.196:6816/588110318 conn(0x7fc63c06e0c0 legacy=0x7fc63c070580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:01.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.467+0000 7fc655b6d640 10 monclient: shutdown 2024-03-29T11:31:01.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.467+0000 7fc655b6d640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:01.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.467+0000 7fc655b6d640 1 -- 172.21.15.86:0/125017478 >> v1:172.21.15.196:6789/0 conn(0x7fc6501422b0 legacy=0x7fc6501defb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:01.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.467+0000 7fc655b6d640 1 -- 172.21.15.86:0/125017478 shutdown_connections 2024-03-29T11:31:01.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.467+0000 7fc655b6d640 1 -- 172.21.15.86:0/125017478 >> 172.21.15.86:0/125017478 conn(0x7fc650066a00 msgr2=0x7fc6500b39c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:01.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.467+0000 7fc655b6d640 1 -- 172.21.15.86:0/125017478 shutdown_connections 2024-03-29T11:31:01.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.467+0000 7fc655b6d640 1 -- 172.21.15.86:0/125017478 wait complete. 2024-03-29T11:31:01.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:01.467+0000 7fc655b6d640 1 librados: shutdown 2024-03-29T11:31:01.477 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:31:08.284 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:08.443 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.449+0000 7f327e3b5640 10 monclient: get_monmap_and_config 2024-03-29T11:31:08.443 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.449+0000 7f327e3b5640 10 monclient: build_initial_monmap 2024-03-29T11:31:08.443 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.450+0000 7f327e3b5640 10 monclient: monmap: 2024-03-29T11:31:08.444 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:31:08.444 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:31:08.444 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:31:08.450993+0000 2024-03-29T11:31:08.444 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:31:08.450993+0000 2024-03-29T11:31:08.444 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:08.444 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:31:08.444 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:31:08.444 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:31:08.444 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:31:08.444 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:31:08.445 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.452+0000 7f327e3b5640 1 Processor -- start 2024-03-29T11:31:08.445 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.452+0000 7f327e3b5640 1 -- start start 2024-03-29T11:31:08.445 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.452+0000 7f327e3b5640 10 monclient: init 2024-03-29T11:31:08.450 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.453+0000 7f327e3b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:08.450 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.453+0000 7f327e3b5640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:31:08.450 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.453+0000 7f327e3b5640 10 monclient(hunting): picked mon.noname-a con 0x7f32700389e0 addr v1:172.21.15.86:6789/0 2024-03-29T11:31:08.450 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.453+0000 7f327e3b5640 10 monclient(hunting): picked mon.noname-c con 0x7f3270038dc0 addr v1:172.21.15.196:6789/0 2024-03-29T11:31:08.451 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.453+0000 7f327e3b5640 10 monclient(hunting): picked mon.noname-b con 0x7f3270030660 addr v1:172.21.15.86:6790/0 2024-03-29T11:31:08.451 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.453+0000 7f327e3b5640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32700c54c0 con 0x7f32700389e0 2024-03-29T11:31:08.451 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.453+0000 7f327e3b5640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3270036430 con 0x7f3270038dc0 2024-03-29T11:31:08.451 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.453+0000 7f327e3b5640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3270039840 con 0x7f3270030660 2024-03-29T11:31:08.451 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.453+0000 7f327e3b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:08.451 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.453+0000 7f327e3b5640 10 monclient(hunting): authenticate will time out at 1492.771257s 2024-03-29T11:31:08.451 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.456+0000 7f3277fff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f32700389e0 0x7f3270033520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:39893/0 (socket says 172.21.15.86:39893) 2024-03-29T11:31:08.451 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.456+0000 7f3277fff640 1 -- 172.21.15.86:0/1174810050 learned_addr learned my addr 172.21.15.86:0/1174810050 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:31:08.451 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.456+0000 7f3276ffd640 1 -- 172.21.15.86:0/1174810050 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2971572001 0 0) 0x7f32700c54c0 con 0x7f32700389e0 2024-03-29T11:31:08.451 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.456+0000 7f3276ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:08.451 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.456+0000 7f3276ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:08.451 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.456+0000 7f3276ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:08.451 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.456+0000 7f3276ffd640 10 monclient(hunting): my global_id is 4569 2024-03-29T11:31:08.451 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.456+0000 7f3276ffd640 1 -- 172.21.15.86:0/1174810050 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3268003520 con 0x7f32700389e0 2024-03-29T11:31:08.456 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.461+0000 7f3276ffd640 1 -- 172.21.15.86:0/1174810050 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 674439406 0 0) 0x7f3270036430 con 0x7f3270038dc0 2024-03-29T11:31:08.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.461+0000 7f3276ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:08.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.461+0000 7f3276ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:08.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.461+0000 7f3276ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:08.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.461+0000 7f3276ffd640 10 monclient(hunting): my global_id is 4546 2024-03-29T11:31:08.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.461+0000 7f3276ffd640 1 -- 172.21.15.86:0/1174810050 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f32700c54c0 con 0x7f3270038dc0 2024-03-29T11:31:08.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.461+0000 7f3276ffd640 1 -- 172.21.15.86:0/1174810050 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 842946326 0 0) 0x7f3268003520 con 0x7f32700389e0 2024-03-29T11:31:08.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.461+0000 7f3276ffd640 1 -- 172.21.15.86:0/1174810050 >> v1:172.21.15.86:6790/0 conn(0x7f3270030660 legacy=0x7f32700d3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:08.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.461+0000 7f3276ffd640 1 -- 172.21.15.86:0/1174810050 >> v1:172.21.15.196:6789/0 conn(0x7f3270038dc0 legacy=0x7f32700cadd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:08.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.461+0000 7f3276ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:31:08.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.461+0000 7f3276ffd640 1 monclient: found mon.noname-a 2024-03-29T11:31:08.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.461+0000 7f3276ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:08.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.461+0000 7f3276ffd640 10 monclient: _send_mon_message to mon.noname-a at v1:172.21.15.86:6789/0 2024-03-29T11:31:08.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.461+0000 7f3276ffd640 1 -- 172.21.15.86:0/1174810050 --> v1:172.21.15.86:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3270035690 con 0x7f32700389e0 2024-03-29T11:31:08.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.461+0000 7f3276ffd640 10 monclient: _finish_auth 0 2024-03-29T11:31:08.457 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.461+0000 7f3276ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:31:08.458 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.461+0000 7f3276ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:08.458 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.462+0000 7f327e3b5640 5 monclient: authenticate success, global_id 4569 2024-03-29T11:31:08.458 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.462+0000 7f327e3b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:08.458 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.462+0000 7f3276ffd640 1 -- 172.21.15.86:0/1174810050 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f327804f730 con 0x7f32700389e0 2024-03-29T11:31:08.458 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.462+0000 7f3276ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:08.458 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.462+0000 7f3276ffd640 10 monclient: got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:31:08.458 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.462+0000 7f3276ffd640 10 monclient: dump: 2024-03-29T11:31:08.458 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:31:08.458 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:31:08.458 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:08.458 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:08.458 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:08.458 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:31:08.458 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:31:08.459 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:31:08.459 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:31:08.459 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:31:08.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.462+0000 7f327e3b5640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:08.459 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.464+0000 7f3276ffd640 1 -- 172.21.15.86:0/1174810050 <== mon.0 v1:172.21.15.86:6789/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f3278075f10 con 0x7f32700389e0 2024-03-29T11:31:08.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.464+0000 7f3276ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:08.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.464+0000 7f3276ffd640 1 -- 172.21.15.86:0/1174810050 <== mon.0 v1:172.21.15.86:6789/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f32780758b0 con 0x7f32700389e0 2024-03-29T11:31:08.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.464+0000 7f3276ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:08.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.464+0000 7f3276ffd640 10 monclient: got monmap 1 from mon.a (according to old e1) 2024-03-29T11:31:08.460 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.464+0000 7f3276ffd640 10 monclient: dump: 2024-03-29T11:31:08.460 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:31:08.460 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:31:08.460 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:08.460 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:08.460 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:08.460 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:31:08.460 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:31:08.460 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:31:08.460 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:31:08.460 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:31:08.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.464+0000 7f327e3b5640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:08.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.464+0000 7f327e3b5640 4 set_mon_vals no callback set 2024-03-29T11:31:08.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.464+0000 7f327e3b5640 10 monclient: shutdown 2024-03-29T11:31:08.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.464+0000 7f327e3b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:08.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.464+0000 7f327e3b5640 1 -- 172.21.15.86:0/1174810050 >> v1:172.21.15.86:6789/0 conn(0x7f32700389e0 legacy=0x7f3270033520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:08.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.465+0000 7f327e3b5640 1 -- 172.21.15.86:0/1174810050 shutdown_connections 2024-03-29T11:31:08.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.465+0000 7f327e3b5640 1 -- 172.21.15.86:0/1174810050 >> 172.21.15.86:0/1174810050 conn(0x7f3270005c00 msgr2=0x7f3270006010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:08.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.465+0000 7f327e3b5640 1 -- 172.21.15.86:0/1174810050 shutdown_connections 2024-03-29T11:31:08.461 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.468+0000 7f327e3b5640 1 -- 172.21.15.86:0/1174810050 wait complete. 2024-03-29T11:31:08.463 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.469+0000 7f327e3b5640 10 monclient: build_initial_monmap 2024-03-29T11:31:08.463 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.469+0000 7f327e3b5640 10 monclient: monmap: 2024-03-29T11:31:08.463 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:31:08.463 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:08.463 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:31:08.463 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:31:08.463 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:08.463 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:31:08.464 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:31:08.464 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:31:08.464 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:31:08.464 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:31:08.464 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.469+0000 7f327e3b5640 1 librados: starting msgr at 2024-03-29T11:31:08.464 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.470+0000 7f327e3b5640 1 librados: starting objecter 2024-03-29T11:31:08.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.470+0000 7f327e3b5640 1 Processor -- start 2024-03-29T11:31:08.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.470+0000 7f327e3b5640 1 -- start start 2024-03-29T11:31:08.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.470+0000 7f327e3b5640 1 librados: setting wanted keys 2024-03-29T11:31:08.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.470+0000 7f327e3b5640 1 librados: calling monclient init 2024-03-29T11:31:08.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.470+0000 7f327e3b5640 10 monclient: init 2024-03-29T11:31:08.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.471+0000 7f327e3b5640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:08.466 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.471+0000 7f327e3b5640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:31:08.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.473+0000 7f327e3b5640 10 monclient(hunting): picked mon.noname-b con 0x7f3270030660 addr v1:172.21.15.196:6789/0 2024-03-29T11:31:08.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.474+0000 7f3277fff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f3270030660 0x7f327000bf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:38891/0 (socket says 172.21.15.86:38891) 2024-03-29T11:31:08.467 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.474+0000 7f327e3b5640 10 monclient(hunting): picked mon.noname-c con 0x7f32700389e0 addr v1:172.21.15.86:6790/0 2024-03-29T11:31:08.471 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.477+0000 7f327e3b5640 10 monclient(hunting): picked mon.noname-a con 0x7f327000c670 addr v1:172.21.15.86:6789/0 2024-03-29T11:31:08.471 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.477+0000 7f327e3b5640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32701642a0 con 0x7f327000c670 2024-03-29T11:31:08.471 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.477+0000 7f327e3b5640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f327000a670 con 0x7f3270030660 2024-03-29T11:31:08.471 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.477+0000 7f327e3b5640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f327002f160 con 0x7f32700389e0 2024-03-29T11:31:08.471 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.477+0000 7f327e3b5640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:08.471 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.477+0000 7f327e3b5640 10 monclient(hunting): authenticate will time out at 1492.795770s 2024-03-29T11:31:08.471 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.477+0000 7f3277fff640 1 -- 172.21.15.86:0/3944932726 learned_addr learned my addr 172.21.15.86:0/3944932726 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:31:08.476 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.481+0000 7f3274ff9640 10 client.?.objecter ms_handle_connect 0x7f3270030660 2024-03-29T11:31:08.476 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.481+0000 7f3274ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:08.476 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.481+0000 7f3274ff9640 10 client.?.objecter ms_handle_connect 0x7f327000c670 2024-03-29T11:31:08.476 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.481+0000 7f3274ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:08.476 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.481+0000 7f3274ff9640 10 client.?.objecter ms_handle_connect 0x7f32700389e0 2024-03-29T11:31:08.476 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.481+0000 7f3274ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:08.476 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.481+0000 7f3274ff9640 1 -- 172.21.15.86:0/3944932726 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3456841590 0 0) 0x7f327000a670 con 0x7f3270030660 2024-03-29T11:31:08.476 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.481+0000 7f3274ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:31:08.476 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.481+0000 7f3274ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:08.476 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.481+0000 7f3274ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:08.476 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.481+0000 7f3274ff9640 10 monclient(hunting): my global_id is 4549 2024-03-29T11:31:08.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.481+0000 7f3274ff9640 1 -- 172.21.15.86:0/3944932726 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f325c004eb0 con 0x7f3270030660 2024-03-29T11:31:08.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.481+0000 7f3274ff9640 1 -- 172.21.15.86:0/3944932726 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1164930500 0 0) 0x7f327002f160 con 0x7f32700389e0 2024-03-29T11:31:08.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.481+0000 7f3274ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:31:08.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.481+0000 7f3274ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:08.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.481+0000 7f3274ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:08.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.481+0000 7f3274ff9640 10 monclient(hunting): my global_id is 4679 2024-03-29T11:31:08.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.481+0000 7f3274ff9640 1 -- 172.21.15.86:0/3944932726 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f327000a670 con 0x7f32700389e0 2024-03-29T11:31:08.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.481+0000 7f3274ff9640 1 -- 172.21.15.86:0/3944932726 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1720026038 0 0) 0x7f32701642a0 con 0x7f327000c670 2024-03-29T11:31:08.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.481+0000 7f3274ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:31:08.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.481+0000 7f3274ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:08.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.481+0000 7f3274ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:08.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.481+0000 7f3274ff9640 10 monclient(hunting): my global_id is 4572 2024-03-29T11:31:08.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.481+0000 7f3274ff9640 1 -- 172.21.15.86:0/3944932726 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f327002f160 con 0x7f327000c670 2024-03-29T11:31:08.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.481+0000 7f3274ff9640 1 -- 172.21.15.86:0/3944932726 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2376782814 0 0) 0x7f325c004eb0 con 0x7f3270030660 2024-03-29T11:31:08.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.481+0000 7f3274ff9640 1 -- 172.21.15.86:0/3944932726 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f32701642a0 con 0x7f3270030660 2024-03-29T11:31:08.477 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.481+0000 7f3274ff9640 1 -- 172.21.15.86:0/3944932726 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f327804f400 con 0x7f3270030660 2024-03-29T11:31:08.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.481+0000 7f3274ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:08.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.481+0000 7f3274ff9640 10 monclient(hunting): got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:31:08.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.481+0000 7f3274ff9640 10 monclient(hunting): dump: 2024-03-29T11:31:08.478 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:31:08.478 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:31:08.478 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:08.478 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:08.478 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:08.478 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:31:08.478 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:31:08.478 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:31:08.478 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:31:08.478 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:31:08.478 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.481+0000 7f3274ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:08.479 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.482+0000 7f3274ff9640 1 -- 172.21.15.86:0/3944932726 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 291969439 0 0) 0x7f327000a670 con 0x7f32700389e0 2024-03-29T11:31:08.479 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.482+0000 7f3274ff9640 1 -- 172.21.15.86:0/3944932726 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f325c004eb0 con 0x7f32700389e0 2024-03-29T11:31:08.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.482+0000 7f3274ff9640 1 -- 172.21.15.86:0/3944932726 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f326c001750 con 0x7f32700389e0 2024-03-29T11:31:08.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.485+0000 7f3274ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:08.481 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.485+0000 7f3274ff9640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:08.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.485+0000 7f3274ff9640 10 monclient(hunting): dump: 2024-03-29T11:31:08.482 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:31:08.482 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:31:08.482 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:08.482 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:08.482 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:08.482 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:31:08.482 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:31:08.482 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:31:08.482 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:31:08.482 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:31:08.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.485+0000 7f3274ff9640 1 -- 172.21.15.86:0/3944932726 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2500122992 0 0) 0x7f327002f160 con 0x7f327000c670 2024-03-29T11:31:08.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.485+0000 7f3274ff9640 1 -- 172.21.15.86:0/3944932726 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f327000a670 con 0x7f327000c670 2024-03-29T11:31:08.482 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.485+0000 7f3274ff9640 1 -- 172.21.15.86:0/3944932726 <== mon.1 v1:172.21.15.196:6789/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1769719309 0 0) 0x7f32701642a0 con 0x7f3270030660 2024-03-29T11:31:08.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.485+0000 7f3274ff9640 1 -- 172.21.15.86:0/3944932726 >> v1:172.21.15.86:6790/0 conn(0x7f32700389e0 legacy=0x7f327002f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:08.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.485+0000 7f3274ff9640 1 -- 172.21.15.86:0/3944932726 >> v1:172.21.15.86:6789/0 conn(0x7f327000c670 legacy=0x7f327016c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:08.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.485+0000 7f3274ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:31:08.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.485+0000 7f3274ff9640 1 monclient: found mon.b 2024-03-29T11:31:08.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.485+0000 7f3274ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:08.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.485+0000 7f3274ff9640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:31:08.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.485+0000 7f3274ff9640 1 -- 172.21.15.86:0/3944932726 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f327002ec40 con 0x7f3270030660 2024-03-29T11:31:08.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.485+0000 7f3274ff9640 10 monclient: _finish_auth 0 2024-03-29T11:31:08.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.485+0000 7f3274ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:31:08.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.485+0000 7f3274ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:08.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.485+0000 7f327e3b5640 5 monclient: authenticate success, global_id 4549 2024-03-29T11:31:08.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.485+0000 7f327e3b5640 10 monclient: _renew_subs 2024-03-29T11:31:08.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.485+0000 7f327e3b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:31:08.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.485+0000 7f327e3b5640 1 -- 172.21.15.86:0/3944932726 --> v1:172.21.15.196:6789/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f327002ee90 con 0x7f3270030660 2024-03-29T11:31:08.483 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.486+0000 7f327e3b5640 10 client.4549.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:08.484 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.486+0000 7f327e3b5640 10 monclient: _renew_subs 2024-03-29T11:31:08.484 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.486+0000 7f327e3b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:31:08.484 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.486+0000 7f327e3b5640 1 -- 172.21.15.86:0/3944932726 --> v1:172.21.15.196:6789/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f327000ccc0 con 0x7f3270030660 2024-03-29T11:31:08.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.486+0000 7f327e3b5640 1 librados: init done 2024-03-29T11:31:08.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.486+0000 7f3274ff9640 1 -- 172.21.15.86:0/3944932726 <== mon.1 v1:172.21.15.196:6789/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f327804fdd0 con 0x7f3270030660 2024-03-29T11:31:08.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.486+0000 7f3274ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:08.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.486+0000 7f3274ff9640 1 -- 172.21.15.86:0/3944932726 <== mon.1 v1:172.21.15.196:6789/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f3278067bf0 con 0x7f3270030660 2024-03-29T11:31:08.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.486+0000 7f3274ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:08.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.486+0000 7f3274ff9640 10 monclient: got monmap 1 from mon.b (according to old e1) 2024-03-29T11:31:08.486 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.486+0000 7f3274ff9640 10 monclient: dump: 2024-03-29T11:31:08.486 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:31:08.487 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:31:08.487 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:08.487 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:08.487 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:08.487 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:31:08.487 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:31:08.487 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:31:08.487 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:31:08.487 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:31:08.487 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.489+0000 7f3274ff9640 1 -- 172.21.15.86:0/3944932726 <== mon.1 v1:172.21.15.196:6789/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f3278067d50 con 0x7f3270030660 2024-03-29T11:31:08.487 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.489+0000 7f32777fe640 10 monclient: get_auth_request con 0x7f325c06dfe0 auth_method 0 2024-03-29T11:31:08.487 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.489+0000 7f3274ff9640 1 -- 172.21.15.86:0/3944932726 <== mon.1 v1:172.21.15.196:6789/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2923349858 0 0) 0x7f32780ff550 con 0x7f3270030660 2024-03-29T11:31:08.487 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.489+0000 7f3274ff9640 10 client.4549.objecter ms_dispatch 0x7f3278000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:31:08.487 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.489+0000 7f3274ff9640 3 client.4549.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:31:08.487 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.489+0000 7f3274ff9640 3 client.4549.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:31:08.488 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.489+0000 7f3274ff9640 20 client.4549.objecter dump_active .. 0 homeless 2024-03-29T11:31:08.488 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.490+0000 7f3274ff9640 10 client.4549.objecter ms_handle_connect 0x7f325c06dfe0 2024-03-29T11:31:08.488 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.491+0000 7f32767fc640 4 set_mon_vals no callback set 2024-03-29T11:31:08.488 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.492+0000 7f327e3b5640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:08.488 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.492+0000 7f327e3b5640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:08.488 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.492+0000 7f327e3b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:31:08.488 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.492+0000 7f327e3b5640 1 -- 172.21.15.86:0/3944932726 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3258005080 con 0x7f3270030660 2024-03-29T11:31:08.493 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.498+0000 7f3274ff9640 1 -- 172.21.15.86:0/3944932726 <== mon.1 v1:172.21.15.196:6789/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f32780cae30 con 0x7f3270030660 2024-03-29T11:31:08.494 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.498+0000 7f3274ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:08.494 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.498+0000 7f3274ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:08.834 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.840+0000 7f327e3b5640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:08.834 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.840+0000 7f327e3b5640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:08.834 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.840+0000 7f327e3b5640 10 monclient: _send_mon_message to mon.b at v1:172.21.15.196:6789/0 2024-03-29T11:31:08.834 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.840+0000 7f327e3b5640 1 -- 172.21.15.86:0/3944932726 --> v1:172.21.15.196:6789/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3258005330 con 0x7f3270030660 2024-03-29T11:31:08.835 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.841+0000 7f3274ff9640 1 -- 172.21.15.86:0/3944932726 <== mon.1 v1:172.21.15.196:6789/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3394516648) 0x7f32780ccd60 con 0x7f3270030660 2024-03-29T11:31:08.835 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.841+0000 7f3274ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:08.835 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.841+0000 7f3274ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:08.835 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:31:08.836 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":11,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:30:38.448721+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:30:38.448721+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:08.839 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.845+0000 7f327e3b5640 10 librados: watch_flush enter 2024-03-29T11:31:08.839 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.846+0000 7f327e3b5640 10 librados: watch_flush exit 2024-03-29T11:31:08.839 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.846+0000 7f327e3b5640 20 client.4549.objecter shutdown clearing up homeless session... 2024-03-29T11:31:08.840 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.846+0000 7f327e3b5640 10 client.4549.objecter successfully canceled tick 2024-03-29T11:31:08.840 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.846+0000 7f327e3b5640 1 -- 172.21.15.86:0/3944932726 >> v1:172.21.15.196:6816/588110318 conn(0x7f325c06dfe0 legacy=0x7f325c0704a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:08.840 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.846+0000 7f327e3b5640 10 monclient: shutdown 2024-03-29T11:31:08.840 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.846+0000 7f327e3b5640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:08.840 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.846+0000 7f327e3b5640 1 -- 172.21.15.86:0/3944932726 >> v1:172.21.15.196:6789/0 conn(0x7f3270030660 legacy=0x7f327000bf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:08.840 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.846+0000 7f327e3b5640 1 -- 172.21.15.86:0/3944932726 shutdown_connections 2024-03-29T11:31:08.841 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.846+0000 7f327e3b5640 1 -- 172.21.15.86:0/3944932726 >> 172.21.15.86:0/3944932726 conn(0x7f3270005c00 msgr2=0x7f3270034090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:08.841 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.846+0000 7f327e3b5640 1 -- 172.21.15.86:0/3944932726 shutdown_connections 2024-03-29T11:31:08.841 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.846+0000 7f327e3b5640 1 -- 172.21.15.86:0/3944932726 wait complete. 2024-03-29T11:31:08.841 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:08.846+0000 7f327e3b5640 1 librados: shutdown 2024-03-29T11:31:08.858 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:31:15.665 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:15.821 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.827+0000 7f29b73e6640 10 monclient: get_monmap_and_config 2024-03-29T11:31:15.821 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.827+0000 7f29b73e6640 10 monclient: build_initial_monmap 2024-03-29T11:31:15.821 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.827+0000 7f29b73e6640 10 monclient: monmap: 2024-03-29T11:31:15.821 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:31:15.822 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:31:15.822 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:31:15.828596+0000 2024-03-29T11:31:15.822 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:31:15.828596+0000 2024-03-29T11:31:15.822 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:15.822 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:31:15.822 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:31:15.822 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:31:15.822 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:31:15.822 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:31:15.823 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.828+0000 7f29b73e6640 1 Processor -- start 2024-03-29T11:31:15.824 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.828+0000 7f29b73e6640 1 -- start start 2024-03-29T11:31:15.824 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.828+0000 7f29b73e6640 10 monclient: init 2024-03-29T11:31:15.824 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.829+0000 7f29b73e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:15.825 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.829+0000 7f29b73e6640 10 monclient: _add_conns ranks=[2,0,1] 2024-03-29T11:31:15.826 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.829+0000 7f29b73e6640 10 monclient(hunting): picked mon.noname-b con 0x7f29b00b4860 addr v1:172.21.15.86:6790/0 2024-03-29T11:31:15.826 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.829+0000 7f29b73e6640 10 monclient(hunting): picked mon.noname-a con 0x7f29b00a9b50 addr v1:172.21.15.86:6789/0 2024-03-29T11:31:15.826 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.829+0000 7f29b73e6640 10 monclient(hunting): picked mon.noname-c con 0x7f29b013f7f0 addr v1:172.21.15.196:6789/0 2024-03-29T11:31:15.826 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.829+0000 7f29b73e6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b01453b0 con 0x7f29b00a9b50 2024-03-29T11:31:15.826 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.829+0000 7f29b73e6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b0141e30 con 0x7f29b013f7f0 2024-03-29T11:31:15.826 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.829+0000 7f29b73e6640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b0145fe0 con 0x7f29b00b4860 2024-03-29T11:31:15.826 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.829+0000 7f29b73e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:15.826 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.829+0000 7f29b73e6640 10 monclient(hunting): authenticate will time out at 1500.147652s 2024-03-29T11:31:15.826 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.832+0000 7f29b5182640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f29b00b4860 0x7f29b00b3d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:42469/0 (socket says 172.21.15.86:42469) 2024-03-29T11:31:15.826 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.832+0000 7f29b5182640 1 -- 172.21.15.86:0/2604293001 learned_addr learned my addr 172.21.15.86:0/2604293001 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:31:15.827 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.833+0000 7f29a7fff640 1 -- 172.21.15.86:0/2604293001 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1573988359 0 0) 0x7f29b0145fe0 con 0x7f29b00b4860 2024-03-29T11:31:15.827 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.833+0000 7f29a7fff640 10 monclient(hunting): _negotiate 2024-03-29T11:31:15.829 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.833+0000 7f29a7fff640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:15.829 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.833+0000 7f29a7fff640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:15.829 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.833+0000 7f29a7fff640 10 monclient(hunting): my global_id is 4682 2024-03-29T11:31:15.829 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.833+0000 7f29a7fff640 1 -- 172.21.15.86:0/2604293001 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f299c003520 con 0x7f29b00b4860 2024-03-29T11:31:15.829 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.834+0000 7f29a7fff640 1 -- 172.21.15.86:0/2604293001 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 13633673 0 0) 0x7f299c003520 con 0x7f29b00b4860 2024-03-29T11:31:15.830 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.834+0000 7f29a7fff640 1 -- 172.21.15.86:0/2604293001 >> v1:172.21.15.196:6789/0 conn(0x7f29b013f7f0 legacy=0x7f29b01458d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-29T11:31:15.830 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.834+0000 7f29a7fff640 1 -- 172.21.15.86:0/2604293001 >> v1:172.21.15.86:6789/0 conn(0x7f29b00a9b50 legacy=0x7f29b013f0e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:31:15.834 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.837+0000 7f29a7fff640 10 monclient: _finish_hunting 0 2024-03-29T11:31:15.834 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.837+0000 7f29a7fff640 1 monclient: found mon.noname-b 2024-03-29T11:31:15.834 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.837+0000 7f29a7fff640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:15.834 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.837+0000 7f29a7fff640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:31:15.834 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.837+0000 7f29a7fff640 1 -- 172.21.15.86:0/2604293001 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b0146170 con 0x7f29b00b4860 2024-03-29T11:31:15.834 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.837+0000 7f29a7fff640 10 monclient: _finish_auth 0 2024-03-29T11:31:15.834 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.837+0000 7f29a7fff640 10 monclient: _check_auth_tickets 2024-03-29T11:31:15.834 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.837+0000 7f29a7fff640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:15.834 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.837+0000 7f29a7fff640 1 -- 172.21.15.86:0/2604293001 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f29a8001a10 con 0x7f29b00b4860 2024-03-29T11:31:15.835 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.837+0000 7f29a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:15.835 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.837+0000 7f29a7fff640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:31:15.835 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.837+0000 7f29a7fff640 10 monclient: dump: 2024-03-29T11:31:15.835 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:31:15.835 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:31:15.835 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:15.835 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:15.835 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:15.835 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:31:15.835 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:31:15.835 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:31:15.835 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:31:15.835 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:31:15.836 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.837+0000 7f29b73e6640 5 monclient: authenticate success, global_id 4682 2024-03-29T11:31:15.836 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.837+0000 7f29b73e6640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:15.836 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.837+0000 7f29a7fff640 1 -- 172.21.15.86:0/2604293001 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29a8001be0 con 0x7f29b00b4860 2024-03-29T11:31:15.836 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.837+0000 7f29a7fff640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:15.836 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.837+0000 7f29a7fff640 1 -- 172.21.15.86:0/2604293001 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f29a8002030 con 0x7f29b00b4860 2024-03-29T11:31:15.836 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.837+0000 7f29a7fff640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:15.836 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.837+0000 7f29a7fff640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:15.836 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.837+0000 7f29a7fff640 10 monclient: dump: 2024-03-29T11:31:15.836 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:31:15.836 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:31:15.836 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:15.836 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:15.836 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:15.837 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:31:15.837 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:31:15.837 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:31:15.837 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:31:15.837 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:31:15.837 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.837+0000 7f29b73e6640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:15.837 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.837+0000 7f29b73e6640 4 set_mon_vals no callback set 2024-03-29T11:31:15.837 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.839+0000 7f29b73e6640 10 monclient: shutdown 2024-03-29T11:31:15.837 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.839+0000 7f29b73e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:15.837 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.839+0000 7f29b73e6640 1 -- 172.21.15.86:0/2604293001 >> v1:172.21.15.86:6790/0 conn(0x7f29b00b4860 legacy=0x7f29b00b3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:15.837 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.839+0000 7f29b73e6640 1 -- 172.21.15.86:0/2604293001 shutdown_connections 2024-03-29T11:31:15.837 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.839+0000 7f29b73e6640 1 -- 172.21.15.86:0/2604293001 >> 172.21.15.86:0/2604293001 conn(0x7f29b006d0b0 msgr2=0x7f29b0066a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:15.839 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.841+0000 7f29b73e6640 1 -- 172.21.15.86:0/2604293001 shutdown_connections 2024-03-29T11:31:15.839 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.842+0000 7f29b73e6640 1 -- 172.21.15.86:0/2604293001 wait complete. 2024-03-29T11:31:15.839 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.842+0000 7f29b73e6640 10 monclient: build_initial_monmap 2024-03-29T11:31:15.839 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.842+0000 7f29b73e6640 10 monclient: monmap: 2024-03-29T11:31:15.839 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:31:15.839 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:15.839 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:31:15.839 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:31:15.839 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:15.839 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:31:15.839 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:31:15.840 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:31:15.840 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:31:15.840 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:31:15.840 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.843+0000 7f29b73e6640 1 librados: starting msgr at 2024-03-29T11:31:15.840 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.843+0000 7f29b73e6640 1 librados: starting objecter 2024-03-29T11:31:15.840 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.843+0000 7f29b73e6640 1 Processor -- start 2024-03-29T11:31:15.840 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.843+0000 7f29b73e6640 1 -- start start 2024-03-29T11:31:15.840 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.843+0000 7f29b73e6640 1 librados: setting wanted keys 2024-03-29T11:31:15.840 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.843+0000 7f29b73e6640 1 librados: calling monclient init 2024-03-29T11:31:15.840 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.843+0000 7f29b73e6640 10 monclient: init 2024-03-29T11:31:15.841 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.844+0000 7f29b73e6640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:15.841 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.844+0000 7f29b73e6640 10 monclient: _add_conns ranks=[1,0,2] 2024-03-29T11:31:15.842 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.844+0000 7f29b73e6640 10 monclient(hunting): picked mon.noname-b con 0x7f29b00a9b50 addr v1:172.21.15.196:6789/0 2024-03-29T11:31:15.842 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.844+0000 7f29b73e6640 10 monclient(hunting): picked mon.noname-a con 0x7f29b00b4860 addr v1:172.21.15.86:6789/0 2024-03-29T11:31:15.842 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.844+0000 7f29b73e6640 10 monclient(hunting): picked mon.noname-c con 0x7f29b013f7f0 addr v1:172.21.15.86:6790/0 2024-03-29T11:31:15.842 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.844+0000 7f29b73e6640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b01435b0 con 0x7f29b00b4860 2024-03-29T11:31:15.842 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.844+0000 7f29b73e6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b01366b0 con 0x7f29b00a9b50 2024-03-29T11:31:15.842 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.844+0000 7f29b73e6640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29b0144610 con 0x7f29b013f7f0 2024-03-29T11:31:15.842 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.844+0000 7f29b73e6640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:15.843 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.844+0000 7f29b73e6640 10 monclient(hunting): authenticate will time out at 1500.162644s 2024-03-29T11:31:15.843 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.847+0000 7f29b5182640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f29b00a9b50 0x7f29b013fc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.86:57585/0 (socket says 172.21.15.86:57585) 2024-03-29T11:31:15.843 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.847+0000 7f29b5182640 1 -- 172.21.15.86:0/45504831 learned_addr learned my addr 172.21.15.86:0/45504831 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:31:15.844 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.849+0000 7f29a5ffb640 10 client.?.objecter ms_handle_connect 0x7f29b00a9b50 2024-03-29T11:31:15.844 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.849+0000 7f29a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:15.844 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.849+0000 7f29a5ffb640 10 client.?.objecter ms_handle_connect 0x7f29b00b4860 2024-03-29T11:31:15.844 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.849+0000 7f29a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:15.844 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.849+0000 7f29a5ffb640 1 -- 172.21.15.86:0/45504831 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1080727038 0 0) 0x7f29b01435b0 con 0x7f29b00b4860 2024-03-29T11:31:15.844 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.849+0000 7f29a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:31:15.844 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.849+0000 7f29a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:15.844 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.849+0000 7f29a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:15.848 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.849+0000 7f29a5ffb640 10 monclient(hunting): my global_id is 4575 2024-03-29T11:31:15.848 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.849+0000 7f29a5ffb640 1 -- 172.21.15.86:0/45504831 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2994004eb0 con 0x7f29b00b4860 2024-03-29T11:31:15.848 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.850+0000 7f29a5ffb640 1 -- 172.21.15.86:0/45504831 <== mon.0 v1:172.21.15.86:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1110128106 0 0) 0x7f2994004eb0 con 0x7f29b00b4860 2024-03-29T11:31:15.848 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.850+0000 7f29a5ffb640 1 -- 172.21.15.86:0/45504831 --> v1:172.21.15.86:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29b01435b0 con 0x7f29b00b4860 2024-03-29T11:31:15.848 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.850+0000 7f29a5ffb640 1 -- 172.21.15.86:0/45504831 <== mon.0 v1:172.21.15.86:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f29a0001510 con 0x7f29b00b4860 2024-03-29T11:31:15.848 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.850+0000 7f29a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:15.848 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.850+0000 7f29a5ffb640 10 monclient(hunting): got monmap 1 from mon.noname-a (according to old e1) 2024-03-29T11:31:15.848 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.850+0000 7f29a5ffb640 10 monclient(hunting): dump: 2024-03-29T11:31:15.848 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:31:15.848 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:31:15.848 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:15.849 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:15.849 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:15.849 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:31:15.849 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:31:15.849 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:31:15.849 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:31:15.849 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:31:15.849 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.850+0000 7f29a5ffb640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:15.849 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.850+0000 7f29a5ffb640 10 client.?.objecter ms_handle_connect 0x7f29b013f7f0 2024-03-29T11:31:15.849 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.850+0000 7f29a5ffb640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:15.849 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.850+0000 7f29a5ffb640 1 -- 172.21.15.86:0/45504831 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1255032167 0 0) 0x7f29b0144610 con 0x7f29b013f7f0 2024-03-29T11:31:15.849 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.850+0000 7f29a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:31:15.849 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.850+0000 7f29a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:15.850 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.850+0000 7f29a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:15.850 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.850+0000 7f29a5ffb640 10 monclient(hunting): my global_id is 4685 2024-03-29T11:31:15.850 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.850+0000 7f29a5ffb640 1 -- 172.21.15.86:0/45504831 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2994004eb0 con 0x7f29b013f7f0 2024-03-29T11:31:15.850 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.851+0000 7f29a5ffb640 1 -- 172.21.15.86:0/45504831 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 987037583 0 0) 0x7f29b01366b0 con 0x7f29b00a9b50 2024-03-29T11:31:15.850 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.851+0000 7f29a5ffb640 10 monclient(hunting): _negotiate 2024-03-29T11:31:15.850 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.851+0000 7f29a5ffb640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:15.850 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.851+0000 7f29a5ffb640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:15.850 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.851+0000 7f29a5ffb640 10 monclient(hunting): my global_id is 4552 2024-03-29T11:31:15.850 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.851+0000 7f29a5ffb640 1 -- 172.21.15.86:0/45504831 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f29b0144610 con 0x7f29b00a9b50 2024-03-29T11:31:15.850 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.851+0000 7f29a5ffb640 1 -- 172.21.15.86:0/45504831 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 2847403990 0 0) 0x7f2994004eb0 con 0x7f29b013f7f0 2024-03-29T11:31:15.850 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.851+0000 7f29a5ffb640 1 -- 172.21.15.86:0/45504831 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f29b01366b0 con 0x7f29b013f7f0 2024-03-29T11:31:15.850 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.851+0000 7f29a5ffb640 1 -- 172.21.15.86:0/45504831 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f29ac001650 con 0x7f29b013f7f0 2024-03-29T11:31:15.851 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.851+0000 7f29a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:15.851 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.851+0000 7f29a5ffb640 10 monclient(hunting): got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:15.851 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.851+0000 7f29a5ffb640 10 monclient(hunting): dump: 2024-03-29T11:31:15.851 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:31:15.851 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:31:15.851 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:15.851 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:15.851 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:15.851 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:31:15.851 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:31:15.851 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:31:15.851 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:31:15.852 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:31:15.852 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.851+0000 7f29a5ffb640 1 -- 172.21.15.86:0/45504831 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3604725748 0 0) 0x7f29b0144610 con 0x7f29b00a9b50 2024-03-29T11:31:15.852 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.851+0000 7f29a5ffb640 1 -- 172.21.15.86:0/45504831 --> v1:172.21.15.196:6789/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f2994004eb0 con 0x7f29b00a9b50 2024-03-29T11:31:15.852 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.851+0000 7f29a5ffb640 1 -- 172.21.15.86:0/45504831 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f29a8001ef0 con 0x7f29b00a9b50 2024-03-29T11:31:15.852 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.851+0000 7f29a5ffb640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:15.852 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.851+0000 7f29a5ffb640 10 monclient(hunting): got monmap 1 from mon.b (according to old e1) 2024-03-29T11:31:15.852 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.851+0000 7f29a5ffb640 10 monclient(hunting): dump: 2024-03-29T11:31:15.852 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:31:15.852 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:31:15.853 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:15.853 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:15.853 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:15.853 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:31:15.853 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:31:15.853 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:31:15.853 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:31:15.853 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:31:15.853 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.852+0000 7f29a5ffb640 1 -- 172.21.15.86:0/45504831 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 4076793808 0 0) 0x7f29b01366b0 con 0x7f29b013f7f0 2024-03-29T11:31:15.853 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.852+0000 7f29a5ffb640 1 -- 172.21.15.86:0/45504831 >> v1:172.21.15.196:6789/0 conn(0x7f29b00a9b50 legacy=0x7f29b013fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:15.853 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.852+0000 7f29a5ffb640 1 -- 172.21.15.86:0/45504831 >> v1:172.21.15.86:6789/0 conn(0x7f29b00b4860 legacy=0x7f29b0135890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:15.853 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.852+0000 7f29a5ffb640 10 monclient: _finish_hunting 0 2024-03-29T11:31:15.853 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.852+0000 7f29a5ffb640 1 monclient: found mon.c 2024-03-29T11:31:15.854 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.852+0000 7f29a5ffb640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:15.854 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.852+0000 7f29a5ffb640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:31:15.854 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.852+0000 7f29a5ffb640 1 -- 172.21.15.86:0/45504831 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29b01448f0 con 0x7f29b013f7f0 2024-03-29T11:31:15.854 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.852+0000 7f29a5ffb640 10 monclient: _finish_auth 0 2024-03-29T11:31:15.854 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.852+0000 7f29a5ffb640 10 monclient: _check_auth_tickets 2024-03-29T11:31:15.854 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.852+0000 7f29a5ffb640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:15.854 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.852+0000 7f29a5ffb640 1 -- 172.21.15.86:0/45504831 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f29ac001b60 con 0x7f29b013f7f0 2024-03-29T11:31:15.854 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.852+0000 7f29a5ffb640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:15.854 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.852+0000 7f29a77fe640 4 set_mon_vals no callback set 2024-03-29T11:31:15.854 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.853+0000 7f29a5ffb640 1 -- 172.21.15.86:0/45504831 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f29ac003790 con 0x7f29b013f7f0 2024-03-29T11:31:15.854 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.853+0000 7f29a5ffb640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:15.855 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.853+0000 7f29a5ffb640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:15.855 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.853+0000 7f29a5ffb640 10 monclient: dump: 2024-03-29T11:31:15.855 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:31:15.855 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:31:15.855 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:15.855 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:15.855 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:15.855 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:31:15.855 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:31:15.855 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:31:15.855 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:31:15.855 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:31:15.855 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.853+0000 7f29b73e6640 5 monclient: authenticate success, global_id 4685 2024-03-29T11:31:15.856 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.853+0000 7f29b73e6640 10 monclient: _renew_subs 2024-03-29T11:31:15.856 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.853+0000 7f29b73e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:31:15.856 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.853+0000 7f29b73e6640 1 -- 172.21.15.86:0/45504831 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29b01431b0 con 0x7f29b013f7f0 2024-03-29T11:31:15.856 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.854+0000 7f29b73e6640 10 client.4685.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:15.856 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.854+0000 7f29b73e6640 10 monclient: _renew_subs 2024-03-29T11:31:15.856 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.854+0000 7f29b73e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:31:15.856 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.854+0000 7f29b73e6640 1 -- 172.21.15.86:0/45504831 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f29b01451d0 con 0x7f29b013f7f0 2024-03-29T11:31:15.856 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.854+0000 7f29b73e6640 1 librados: init done 2024-03-29T11:31:15.857 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.856+0000 7f29a5ffb640 1 -- 172.21.15.86:0/45504831 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f29ac019760 con 0x7f29b013f7f0 2024-03-29T11:31:15.857 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.857+0000 7f29b73e6640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:15.857 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.857+0000 7f29b73e6640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:15.857 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.857+0000 7f29b73e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:31:15.857 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.857+0000 7f29b73e6640 1 -- 172.21.15.86:0/45504831 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29b0143760 con 0x7f29b013f7f0 2024-03-29T11:31:15.857 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.857+0000 7f29a5ffb640 1 -- 172.21.15.86:0/45504831 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2923349858 0 0) 0x7f29ac083560 con 0x7f29b013f7f0 2024-03-29T11:31:15.857 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.857+0000 7f29a5ffb640 10 client.4685.objecter ms_dispatch 0x7f29b0000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:31:15.857 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.857+0000 7f29a5ffb640 3 client.4685.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:31:15.857 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.857+0000 7f29a5ffb640 3 client.4685.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:31:15.857 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.857+0000 7f29b5182640 10 monclient: get_auth_request con 0x7f299406e110 auth_method 0 2024-03-29T11:31:15.858 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.857+0000 7f29a5ffb640 20 client.4685.objecter dump_active .. 0 homeless 2024-03-29T11:31:15.858 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.859+0000 7f29a5ffb640 10 client.4685.objecter ms_handle_connect 0x7f299406e110 2024-03-29T11:31:15.860 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.866+0000 7f29a5ffb640 1 -- 172.21.15.86:0/45504831 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f29b0143760 con 0x7f29b013f7f0 2024-03-29T11:31:15.860 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.866+0000 7f29a5ffb640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:15.860 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:15.866+0000 7f29a5ffb640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:16.194 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:16.200+0000 7f29b73e6640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:16.194 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:16.200+0000 7f29b73e6640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:16.194 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:16.200+0000 7f29b73e6640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:31:16.194 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:16.200+0000 7f29b73e6640 1 -- 172.21.15.86:0/45504831 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29b0143760 con 0x7f29b013f7f0 2024-03-29T11:31:16.195 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:16.201+0000 7f29a5ffb640 1 -- 172.21.15.86:0/45504831 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3394516648) 0x7f29b0143760 con 0x7f29b013f7f0 2024-03-29T11:31:16.195 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:16.201+0000 7f29a5ffb640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:16.195 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:16.201+0000 7f29a5ffb640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:16.195 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:31:16.195 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":11,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:30:38.448721+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:30:38.448721+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:16.198 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:16.205+0000 7f29b73e6640 10 librados: watch_flush enter 2024-03-29T11:31:16.200 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:16.206+0000 7f29b73e6640 10 librados: watch_flush exit 2024-03-29T11:31:16.200 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:16.206+0000 7f29b73e6640 20 client.4685.objecter shutdown clearing up homeless session... 2024-03-29T11:31:16.200 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:16.206+0000 7f29b73e6640 10 client.4685.objecter successfully canceled tick 2024-03-29T11:31:16.201 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:16.206+0000 7f29b73e6640 1 -- 172.21.15.86:0/45504831 >> v1:172.21.15.196:6816/588110318 conn(0x7f299406e110 legacy=0x7f29940705d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:16.201 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:16.206+0000 7f29b73e6640 10 monclient: shutdown 2024-03-29T11:31:16.201 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:16.206+0000 7f29b73e6640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:16.201 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:16.206+0000 7f29b73e6640 1 -- 172.21.15.86:0/45504831 >> v1:172.21.15.86:6790/0 conn(0x7f29b013f7f0 legacy=0x7f29b0135fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:16.201 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:16.206+0000 7f29b73e6640 1 -- 172.21.15.86:0/45504831 shutdown_connections 2024-03-29T11:31:16.201 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:16.206+0000 7f29b73e6640 1 -- 172.21.15.86:0/45504831 >> 172.21.15.86:0/45504831 conn(0x7f29b006d0b0 msgr2=0x7f29b013eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:16.201 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:16.206+0000 7f29b73e6640 1 -- 172.21.15.86:0/45504831 shutdown_connections 2024-03-29T11:31:16.201 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:16.206+0000 7f29b73e6640 1 -- 172.21.15.86:0/45504831 wait complete. 2024-03-29T11:31:16.201 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:16.207+0000 7f29b73e6640 1 librados: shutdown 2024-03-29T11:31:16.218 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:31:23.026 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:23.172 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.178+0000 7f95a7d1c640 10 monclient: get_monmap_and_config 2024-03-29T11:31:23.172 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.178+0000 7f95a7d1c640 10 monclient: build_initial_monmap 2024-03-29T11:31:23.172 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.178+0000 7f95a7d1c640 10 monclient: monmap: 2024-03-29T11:31:23.172 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:31:23.172 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:31:23.172 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:31:23.179259+0000 2024-03-29T11:31:23.172 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:31:23.179259+0000 2024-03-29T11:31:23.172 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:23.172 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:31:23.173 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:31:23.173 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:31:23.173 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:31:23.173 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:31:23.173 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.179+0000 7f95a7d1c640 1 Processor -- start 2024-03-29T11:31:23.173 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.179+0000 7f95a7d1c640 1 -- start start 2024-03-29T11:31:23.174 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.179+0000 7f95a7d1c640 10 monclient: init 2024-03-29T11:31:23.176 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.180+0000 7f95a7d1c640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:23.176 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.180+0000 7f95a7d1c640 10 monclient: _add_conns ranks=[1,2,0] 2024-03-29T11:31:23.176 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.180+0000 7f95a7d1c640 10 monclient(hunting): picked mon.noname-c con 0x7f95a00b1f60 addr v1:172.21.15.196:6789/0 2024-03-29T11:31:23.176 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.180+0000 7f95a7d1c640 10 monclient(hunting): picked mon.noname-b con 0x7f95a0071eb0 addr v1:172.21.15.86:6790/0 2024-03-29T11:31:23.177 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.180+0000 7f95a7d1c640 10 monclient(hunting): picked mon.noname-a con 0x7f95a0143d10 addr v1:172.21.15.86:6789/0 2024-03-29T11:31:23.177 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.180+0000 7f95a7d1c640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a006ef80 con 0x7f95a0143d10 2024-03-29T11:31:23.177 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.180+0000 7f95a7d1c640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a006f260 con 0x7f95a00b1f60 2024-03-29T11:31:23.177 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.180+0000 7f95a7d1c640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a00b40c0 con 0x7f95a0071eb0 2024-03-29T11:31:23.177 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.180+0000 7f95a7d1c640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:23.177 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.180+0000 7f95a7d1c640 10 monclient(hunting): authenticate will time out at 1507.498577s 2024-03-29T11:31:23.177 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.180+0000 7f95a52b7640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f95a0071eb0 0x7f95a0146470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:60298/0 (socket says 172.21.15.86:60298) 2024-03-29T11:31:23.177 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.180+0000 7f95a52b7640 1 -- 172.21.15.86:0/3676496491 learned_addr learned my addr 172.21.15.86:0/3676496491 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:31:23.177 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.181+0000 7f95a4ab6640 1 -- 172.21.15.86:0/3676496491 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 463487936 0 0) 0x7f95a00b40c0 con 0x7f95a0071eb0 2024-03-29T11:31:23.177 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.181+0000 7f95a4ab6640 10 monclient(hunting): _negotiate 2024-03-29T11:31:23.177 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.181+0000 7f95a4ab6640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:23.177 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.181+0000 7f95a4ab6640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:23.178 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.181+0000 7f95a4ab6640 10 monclient(hunting): my global_id is 4688 2024-03-29T11:31:23.178 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.181+0000 7f95a4ab6640 1 -- 172.21.15.86:0/3676496491 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f958c003520 con 0x7f95a0071eb0 2024-03-29T11:31:23.178 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.181+0000 7f95a4ab6640 1 -- 172.21.15.86:0/3676496491 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1676383147 0 0) 0x7f958c003520 con 0x7f95a0071eb0 2024-03-29T11:31:23.178 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.181+0000 7f95a4ab6640 1 -- 172.21.15.86:0/3676496491 >> v1:172.21.15.196:6789/0 conn(0x7f95a00b1f60 legacy=0x7f95a0143600 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:31:23.182 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.183+0000 7f95a4ab6640 1 -- 172.21.15.86:0/3676496491 >> v1:172.21.15.86:6789/0 conn(0x7f95a0143d10 legacy=0x7f95a0146b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:31:23.182 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.184+0000 7f95a4ab6640 10 monclient: _finish_hunting 0 2024-03-29T11:31:23.182 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.184+0000 7f95a4ab6640 1 monclient: found mon.noname-b 2024-03-29T11:31:23.182 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.184+0000 7f95a4ab6640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:23.182 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.184+0000 7f95a4ab6640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:31:23.182 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.184+0000 7f95a4ab6640 1 -- 172.21.15.86:0/3676496491 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a0155d70 con 0x7f95a0071eb0 2024-03-29T11:31:23.182 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.184+0000 7f95a4ab6640 10 monclient: _finish_auth 0 2024-03-29T11:31:23.182 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.184+0000 7f95a4ab6640 10 monclient: _check_auth_tickets 2024-03-29T11:31:23.182 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.184+0000 7f95a4ab6640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:23.182 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.184+0000 7f95a4ab6640 1 -- 172.21.15.86:0/3676496491 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f959c001690 con 0x7f95a0071eb0 2024-03-29T11:31:23.182 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.184+0000 7f95a4ab6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:23.183 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.184+0000 7f95a4ab6640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:31:23.183 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.184+0000 7f95a4ab6640 10 monclient: dump: 2024-03-29T11:31:23.183 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:31:23.183 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:31:23.183 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:23.183 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:23.183 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:23.183 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:31:23.183 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:31:23.183 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:31:23.183 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:31:23.183 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:31:23.183 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.184+0000 7f95a4ab6640 1 -- 172.21.15.86:0/3676496491 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f959c001920 con 0x7f95a0071eb0 2024-03-29T11:31:23.183 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.184+0000 7f95a4ab6640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:23.183 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.184+0000 7f95a4ab6640 1 -- 172.21.15.86:0/3676496491 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f959c001cb0 con 0x7f95a0071eb0 2024-03-29T11:31:23.184 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.184+0000 7f95a4ab6640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:23.184 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.184+0000 7f95a4ab6640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:23.184 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.184+0000 7f95a4ab6640 10 monclient: dump: 2024-03-29T11:31:23.184 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:31:23.184 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:31:23.184 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:23.184 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:23.184 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:23.184 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:31:23.184 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:31:23.184 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:31:23.184 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:31:23.184 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:31:23.185 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.185+0000 7f95a7d1c640 5 monclient: authenticate success, global_id 4688 2024-03-29T11:31:23.185 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.185+0000 7f95a7d1c640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:23.185 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.185+0000 7f95a7d1c640 4 set_mon_vals no callback set 2024-03-29T11:31:23.185 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.186+0000 7f95a7d1c640 10 monclient: shutdown 2024-03-29T11:31:23.185 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.186+0000 7f95a7d1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:23.185 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.186+0000 7f95a7d1c640 1 -- 172.21.15.86:0/3676496491 >> v1:172.21.15.86:6790/0 conn(0x7f95a0071eb0 legacy=0x7f95a0146470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:23.185 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.186+0000 7f95a7d1c640 1 -- 172.21.15.86:0/3676496491 shutdown_connections 2024-03-29T11:31:23.185 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.186+0000 7f95a7d1c640 1 -- 172.21.15.86:0/3676496491 >> 172.21.15.86:0/3676496491 conn(0x7f95a0066a00 msgr2=0x7f95a0066e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:23.185 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.186+0000 7f95a7d1c640 1 -- 172.21.15.86:0/3676496491 shutdown_connections 2024-03-29T11:31:23.185 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.186+0000 7f95a7d1c640 1 -- 172.21.15.86:0/3676496491 wait complete. 2024-03-29T11:31:23.185 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.187+0000 7f95a7d1c640 10 monclient: build_initial_monmap 2024-03-29T11:31:23.185 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.187+0000 7f95a7d1c640 10 monclient: monmap: 2024-03-29T11:31:23.185 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:31:23.186 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:23.186 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:31:23.186 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:31:23.186 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:23.186 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:31:23.186 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:31:23.186 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:31:23.186 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:31:23.186 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:31:23.186 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.188+0000 7f95a7d1c640 1 librados: starting msgr at 2024-03-29T11:31:23.186 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.188+0000 7f95a7d1c640 1 librados: starting objecter 2024-03-29T11:31:23.186 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.188+0000 7f95a7d1c640 1 Processor -- start 2024-03-29T11:31:23.186 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.188+0000 7f95a7d1c640 1 -- start start 2024-03-29T11:31:23.187 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.188+0000 7f95a7d1c640 1 librados: setting wanted keys 2024-03-29T11:31:23.187 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.188+0000 7f95a7d1c640 1 librados: calling monclient init 2024-03-29T11:31:23.187 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.188+0000 7f95a7d1c640 10 monclient: init 2024-03-29T11:31:23.193 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.188+0000 7f95a7d1c640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:23.193 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.188+0000 7f95a7d1c640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:31:23.193 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.188+0000 7f95a7d1c640 10 monclient(hunting): picked mon.noname-c con 0x7f95a00b1f60 addr v1:172.21.15.86:6790/0 2024-03-29T11:31:23.193 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.188+0000 7f95a7d1c640 10 monclient(hunting): picked mon.noname-b con 0x7f95a0143d10 addr v1:172.21.15.196:6789/0 2024-03-29T11:31:23.193 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.189+0000 7f95a7d1c640 10 monclient(hunting): picked mon.noname-a con 0x7f95a0070750 addr v1:172.21.15.86:6789/0 2024-03-29T11:31:23.193 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.189+0000 7f95a7d1c640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a0154bb0 con 0x7f95a0070750 2024-03-29T11:31:23.193 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.189+0000 7f95a7d1c640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a0139400 con 0x7f95a0143d10 2024-03-29T11:31:23.193 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.189+0000 7f95a7d1c640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a0070b50 con 0x7f95a00b1f60 2024-03-29T11:31:23.194 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.189+0000 7f95a7d1c640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:23.194 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.189+0000 7f95a7d1c640 10 monclient(hunting): authenticate will time out at 1507.507042s 2024-03-29T11:31:23.194 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.189+0000 7f95a5ab8640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f95a00b1f60 0x7f95a0154370 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:34578/0 (socket says 172.21.15.86:34578) 2024-03-29T11:31:23.194 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.189+0000 7f95a5ab8640 1 -- 172.21.15.86:0/1577402139 learned_addr learned my addr 172.21.15.86:0/1577402139 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:31:23.194 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.189+0000 7f95967fc640 10 client.?.objecter ms_handle_connect 0x7f95a00b1f60 2024-03-29T11:31:23.194 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.189+0000 7f95967fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:23.194 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.189+0000 7f95967fc640 1 -- 172.21.15.86:0/1577402139 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3831875696 0 0) 0x7f95a0070b50 con 0x7f95a00b1f60 2024-03-29T11:31:23.194 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.189+0000 7f95967fc640 10 monclient(hunting): _negotiate 2024-03-29T11:31:23.194 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.189+0000 7f95967fc640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:23.195 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.189+0000 7f95967fc640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:23.195 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.189+0000 7f95967fc640 10 monclient(hunting): my global_id is 4691 2024-03-29T11:31:23.195 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.189+0000 7f95967fc640 1 -- 172.21.15.86:0/1577402139 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9584004eb0 con 0x7f95a00b1f60 2024-03-29T11:31:23.195 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.190+0000 7f95967fc640 1 -- 172.21.15.86:0/1577402139 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 3057897924 0 0) 0x7f9584004eb0 con 0x7f95a00b1f60 2024-03-29T11:31:23.195 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.190+0000 7f95967fc640 1 -- 172.21.15.86:0/1577402139 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f9584005300 con 0x7f95a00b1f60 2024-03-29T11:31:23.195 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.190+0000 7f95967fc640 1 -- 172.21.15.86:0/1577402139 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f9598001bb0 con 0x7f95a00b1f60 2024-03-29T11:31:23.195 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.190+0000 7f95967fc640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:23.195 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.190+0000 7f95967fc640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:31:23.195 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.190+0000 7f95967fc640 10 monclient(hunting): dump: 2024-03-29T11:31:23.195 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:31:23.195 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:31:23.195 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:23.195 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:23.196 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:23.196 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:31:23.196 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:31:23.196 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:31:23.196 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:31:23.196 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:31:23.196 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.190+0000 7f95967fc640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:23.196 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.191+0000 7f95967fc640 1 -- 172.21.15.86:0/1577402139 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 1457437963 0 0) 0x7f9584005300 con 0x7f95a00b1f60 2024-03-29T11:31:23.196 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.191+0000 7f95967fc640 1 -- 172.21.15.86:0/1577402139 >> v1:172.21.15.196:6789/0 conn(0x7f95a0143d10 legacy=0x7f95a0139de0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:31:23.196 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.193+0000 7f95967fc640 1 -- 172.21.15.86:0/1577402139 >> v1:172.21.15.86:6789/0 conn(0x7f95a0070750 legacy=0x7f95a01e5f10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:31:23.196 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.193+0000 7f95967fc640 10 monclient: _finish_hunting 0 2024-03-29T11:31:23.196 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.193+0000 7f95967fc640 1 monclient: found mon.c 2024-03-29T11:31:23.196 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.193+0000 7f95967fc640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:23.196 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.193+0000 7f95967fc640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:31:23.197 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.193+0000 7f95967fc640 1 -- 172.21.15.86:0/1577402139 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a0139670 con 0x7f95a00b1f60 2024-03-29T11:31:23.197 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.193+0000 7f95967fc640 10 monclient: _finish_auth 0 2024-03-29T11:31:23.197 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.193+0000 7f95967fc640 10 monclient: _check_auth_tickets 2024-03-29T11:31:23.197 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.193+0000 7f95967fc640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:23.197 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.193+0000 7f95967fc640 10 client.?.objecter ms_handle_connect 0x7f95a0070750 2024-03-29T11:31:23.197 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.194+0000 7f95967fc640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:23.197 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.194+0000 7f95a7d1c640 5 monclient: authenticate success, global_id 4691 2024-03-29T11:31:23.197 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.194+0000 7f95a7d1c640 10 monclient: _renew_subs 2024-03-29T11:31:23.197 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.194+0000 7f95a7d1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:31:23.197 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.194+0000 7f95a7d1c640 1 -- 172.21.15.86:0/1577402139 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95a0139950 con 0x7f95a00b1f60 2024-03-29T11:31:23.197 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.194+0000 7f95a7d1c640 10 client.4691.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:23.197 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.194+0000 7f95a7d1c640 10 monclient: _renew_subs 2024-03-29T11:31:23.197 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.194+0000 7f95a7d1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:31:23.197 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.194+0000 7f95a7d1c640 1 -- 172.21.15.86:0/1577402139 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f95a01539b0 con 0x7f95a00b1f60 2024-03-29T11:31:23.197 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.194+0000 7f95a7d1c640 1 librados: init done 2024-03-29T11:31:23.198 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.195+0000 7f95967fc640 1 -- 172.21.15.86:0/1577402139 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f9598002180 con 0x7f95a00b1f60 2024-03-29T11:31:23.198 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.195+0000 7f95967fc640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:23.198 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.195+0000 7f95977fe640 4 set_mon_vals no callback set 2024-03-29T11:31:23.198 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.196+0000 7f95967fc640 1 -- 172.21.15.86:0/1577402139 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f9598001bb0 con 0x7f95a00b1f60 2024-03-29T11:31:23.198 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.196+0000 7f95967fc640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:23.198 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.196+0000 7f95967fc640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:23.198 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.196+0000 7f95967fc640 10 monclient: dump: 2024-03-29T11:31:23.198 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:31:23.198 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:31:23.198 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:23.198 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:23.198 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:23.198 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:31:23.199 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:31:23.199 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:31:23.199 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:31:23.199 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:31:23.199 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.197+0000 7f95a7d1c640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:23.199 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.198+0000 7f95967fc640 1 -- 172.21.15.86:0/1577402139 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f959801ab70 con 0x7f95a00b1f60 2024-03-29T11:31:23.199 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.198+0000 7f95967fc640 1 -- 172.21.15.86:0/1577402139 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2923349858 0 0) 0x7f9598084ae0 con 0x7f95a00b1f60 2024-03-29T11:31:23.199 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.198+0000 7f95967fc640 10 client.4691.objecter ms_dispatch 0x7f95a0000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:31:23.199 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.198+0000 7f95967fc640 3 client.4691.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:31:23.199 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.198+0000 7f95967fc640 3 client.4691.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:31:23.199 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.198+0000 7f95967fc640 20 client.4691.objecter dump_active .. 0 homeless 2024-03-29T11:31:23.199 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.199+0000 7f95a7d1c640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:23.199 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.199+0000 7f95a7d1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:31:23.199 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.199+0000 7f95a7d1c640 1 -- 172.21.15.86:0/1577402139 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9588005080 con 0x7f95a00b1f60 2024-03-29T11:31:23.200 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.199+0000 7f95a52b7640 10 monclient: get_auth_request con 0x7f958406df90 auth_method 0 2024-03-29T11:31:23.200 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.199+0000 7f95967fc640 10 client.4691.objecter ms_handle_connect 0x7f958406df90 2024-03-29T11:31:23.202 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.208+0000 7f95967fc640 1 -- 172.21.15.86:0/1577402139 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f9598050420 con 0x7f95a00b1f60 2024-03-29T11:31:23.203 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.208+0000 7f95967fc640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:23.203 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.208+0000 7f95967fc640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:23.536 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.542+0000 7f95a7d1c640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:23.536 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.542+0000 7f95a7d1c640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:23.536 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.542+0000 7f95a7d1c640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:31:23.537 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.542+0000 7f95a7d1c640 1 -- 172.21.15.86:0/1577402139 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9588005330 con 0x7f95a00b1f60 2024-03-29T11:31:23.537 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.543+0000 7f95967fc640 1 -- 172.21.15.86:0/1577402139 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3394516648) 0x7f9598050420 con 0x7f95a00b1f60 2024-03-29T11:31:23.537 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.543+0000 7f95967fc640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:23.537 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.543+0000 7f95967fc640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:23.537 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:31:23.538 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":11,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:30:38.448721+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:30:38.448721+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:23.541 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.547+0000 7f95a7d1c640 10 librados: watch_flush enter 2024-03-29T11:31:23.541 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.547+0000 7f95a7d1c640 10 librados: watch_flush exit 2024-03-29T11:31:23.541 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.547+0000 7f95a7d1c640 20 client.4691.objecter shutdown clearing up homeless session... 2024-03-29T11:31:23.542 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.547+0000 7f95a7d1c640 10 client.4691.objecter successfully canceled tick 2024-03-29T11:31:23.542 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.547+0000 7f95a7d1c640 1 -- 172.21.15.86:0/1577402139 >> v1:172.21.15.196:6816/588110318 conn(0x7f958406df90 legacy=0x7f9584070450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:23.542 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.547+0000 7f95a7d1c640 10 monclient: shutdown 2024-03-29T11:31:23.542 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.547+0000 7f95a7d1c640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:23.542 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.547+0000 7f95a7d1c640 1 -- 172.21.15.86:0/1577402139 >> v1:172.21.15.86:6790/0 conn(0x7f95a00b1f60 legacy=0x7f95a0154370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:23.542 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.547+0000 7f95a7d1c640 1 -- 172.21.15.86:0/1577402139 shutdown_connections 2024-03-29T11:31:23.542 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.547+0000 7f95a7d1c640 1 -- 172.21.15.86:0/1577402139 >> 172.21.15.86:0/1577402139 conn(0x7f95a0066a00 msgr2=0x7f95a00b5810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:23.543 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.548+0000 7f95a7d1c640 1 -- 172.21.15.86:0/1577402139 shutdown_connections 2024-03-29T11:31:23.543 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.548+0000 7f95a7d1c640 1 -- 172.21.15.86:0/1577402139 wait complete. 2024-03-29T11:31:23.543 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:23.549+0000 7f95a7d1c640 1 librados: shutdown 2024-03-29T11:31:23.560 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:31:30.368 DEBUG:teuthology.orchestra.run.smithi086:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-03-29T11:31:30.508 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.514+0000 7f88ee4d9640 10 monclient: get_monmap_and_config 2024-03-29T11:31:30.508 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.514+0000 7f88ee4d9640 10 monclient: build_initial_monmap 2024-03-29T11:31:30.508 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.514+0000 7f88ee4d9640 10 monclient: monmap: 2024-03-29T11:31:30.508 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:31:30.508 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:31:30.509 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:31:30.515363+0000 2024-03-29T11:31:30.509 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:31:30.515363+0000 2024-03-29T11:31:30.509 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:30.509 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:31:30.509 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:31:30.509 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-c 2024-03-29T11:31:30.509 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-b 2024-03-29T11:31:30.509 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:31:30.510 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.515+0000 7f88ee4d9640 1 Processor -- start 2024-03-29T11:31:30.510 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.515+0000 7f88ee4d9640 1 -- start start 2024-03-29T11:31:30.510 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.515+0000 7f88ee4d9640 10 monclient: init 2024-03-29T11:31:30.510 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.516+0000 7f88ee4d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:30.511 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.516+0000 7f88ee4d9640 10 monclient: _add_conns ranks=[2,1,0] 2024-03-29T11:31:30.511 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.516+0000 7f88ee4d9640 10 monclient(hunting): picked mon.noname-b con 0x7f88e81422b0 addr v1:172.21.15.86:6790/0 2024-03-29T11:31:30.511 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.516+0000 7f88ee4d9640 10 monclient(hunting): picked mon.noname-c con 0x7f88e8142690 addr v1:172.21.15.196:6789/0 2024-03-29T11:31:30.511 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.516+0000 7f88ee4d9640 10 monclient(hunting): picked mon.noname-a con 0x7f88e80b17d0 addr v1:172.21.15.86:6789/0 2024-03-29T11:31:30.511 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.516+0000 7f88ee4d9640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88e8142c80 con 0x7f88e80b17d0 2024-03-29T11:31:30.511 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.516+0000 7f88ee4d9640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88e80aae40 con 0x7f88e8142690 2024-03-29T11:31:30.511 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.516+0000 7f88ee4d9640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88e813fbd0 con 0x7f88e81422b0 2024-03-29T11:31:30.511 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.516+0000 7f88ee4d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:30.511 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.516+0000 7f88ee4d9640 10 monclient(hunting): authenticate will time out at 1514.834482s 2024-03-29T11:31:30.511 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.516+0000 7f88e7fff640 1 --1- >> v1:172.21.15.86:6790/0 conn(0x7f88e81422b0 0x7f88e813f4c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.86:6790/0 says I am v1:172.21.15.86:34194/0 (socket says 172.21.15.86:34194) 2024-03-29T11:31:30.511 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.516+0000 7f88e7fff640 1 -- 172.21.15.86:0/3811020330 learned_addr learned my addr 172.21.15.86:0/3811020330 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:31:30.512 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.517+0000 7f88e6ffd640 1 -- 172.21.15.86:0/3811020330 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1956848645 0 0) 0x7f88e813fbd0 con 0x7f88e81422b0 2024-03-29T11:31:30.512 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.517+0000 7f88e6ffd640 10 monclient(hunting): _negotiate 2024-03-29T11:31:30.512 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.517+0000 7f88e6ffd640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:30.513 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.517+0000 7f88e6ffd640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:30.513 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.517+0000 7f88e6ffd640 10 monclient(hunting): my global_id is 4694 2024-03-29T11:31:30.513 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.517+0000 7f88e6ffd640 1 -- 172.21.15.86:0/3811020330 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88d0003520 con 0x7f88e81422b0 2024-03-29T11:31:30.513 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.517+0000 7f88e6ffd640 1 -- 172.21.15.86:0/3811020330 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1537875101 0 0) 0x7f88d0003520 con 0x7f88e81422b0 2024-03-29T11:31:30.513 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.517+0000 7f88e6ffd640 1 -- 172.21.15.86:0/3811020330 >> v1:172.21.15.196:6789/0 conn(0x7f88e8142690 legacy=0x7f88e80b2590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-29T11:31:30.513 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.519+0000 7f88e6ffd640 1 -- 172.21.15.86:0/3811020330 >> v1:172.21.15.86:6789/0 conn(0x7f88e80b17d0 legacy=0x7f88e80b2ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:30.513 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.519+0000 7f88e6ffd640 10 monclient: _finish_hunting 0 2024-03-29T11:31:30.513 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.519+0000 7f88e6ffd640 1 monclient: found mon.noname-b 2024-03-29T11:31:30.514 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.519+0000 7f88e6ffd640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:30.514 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.519+0000 7f88e6ffd640 10 monclient: _send_mon_message to mon.noname-b at v1:172.21.15.86:6790/0 2024-03-29T11:31:30.514 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.519+0000 7f88e6ffd640 1 -- 172.21.15.86:0/3811020330 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e8151940 con 0x7f88e81422b0 2024-03-29T11:31:30.514 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.519+0000 7f88e6ffd640 10 monclient: _finish_auth 0 2024-03-29T11:31:30.514 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.519+0000 7f88e6ffd640 10 monclient: _check_auth_tickets 2024-03-29T11:31:30.514 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.519+0000 7f88e6ffd640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:30.514 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.519+0000 7f88e6ffd640 1 -- 172.21.15.86:0/3811020330 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f88d8001690 con 0x7f88e81422b0 2024-03-29T11:31:30.514 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.519+0000 7f88e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:30.514 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.519+0000 7f88e6ffd640 10 monclient: got monmap 1 from mon.noname-b (according to old e1) 2024-03-29T11:31:30.514 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.519+0000 7f88e6ffd640 10 monclient: dump: 2024-03-29T11:31:30.514 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:31:30.514 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:31:30.515 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:30.515 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:30.515 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:30.515 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:31:30.515 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:31:30.515 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:31:30.515 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:31:30.515 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:31:30.515 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.519+0000 7f88ee4d9640 5 monclient: authenticate success, global_id 4694 2024-03-29T11:31:30.515 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.519+0000 7f88ee4d9640 20 monclient: get_monmap_and_config waiting for monmap|config 2024-03-29T11:31:30.515 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.519+0000 7f88e6ffd640 1 -- 172.21.15.86:0/3811020330 <== mon.2 v1:172.21.15.86:6790/0 4 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88d80018f0 con 0x7f88e81422b0 2024-03-29T11:31:30.515 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.519+0000 7f88e6ffd640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:30.515 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.519+0000 7f88ee4d9640 10 monclient: get_monmap_and_config success 2024-03-29T11:31:30.515 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.519+0000 7f88ee4d9640 4 set_mon_vals no callback set 2024-03-29T11:31:30.523 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.520+0000 7f88ee4d9640 10 monclient: shutdown 2024-03-29T11:31:30.523 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.520+0000 7f88e6ffd640 1 -- 172.21.15.86:0/3811020330 <== mon.2 v1:172.21.15.86:6790/0 5 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f88d8001c80 con 0x7f88e81422b0 2024-03-29T11:31:30.523 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.520+0000 7f88e6ffd640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:30.523 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.520+0000 7f88e6ffd640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:30.523 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.520+0000 7f88e6ffd640 10 monclient: dump: 2024-03-29T11:31:30.523 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:31:30.524 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:31:30.524 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:30.524 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:30.524 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:30.524 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:31:30.524 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:31:30.524 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:31:30.524 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:31:30.524 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:31:30.524 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.520+0000 7f88ee4d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:30.524 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.520+0000 7f88ee4d9640 1 -- 172.21.15.86:0/3811020330 >> v1:172.21.15.86:6790/0 conn(0x7f88e81422b0 legacy=0x7f88e813f4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:30.524 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.520+0000 7f88ee4d9640 1 -- 172.21.15.86:0/3811020330 shutdown_connections 2024-03-29T11:31:30.524 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.520+0000 7f88ee4d9640 1 -- 172.21.15.86:0/3811020330 >> 172.21.15.86:0/3811020330 conn(0x7f88e8066a00 msgr2=0x7f88e8066de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:30.525 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.521+0000 7f88ee4d9640 1 -- 172.21.15.86:0/3811020330 shutdown_connections 2024-03-29T11:31:30.525 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.521+0000 7f88ee4d9640 1 -- 172.21.15.86:0/3811020330 wait complete. 2024-03-29T11:31:30.525 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.521+0000 7f88ee4d9640 10 monclient: build_initial_monmap 2024-03-29T11:31:30.525 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.521+0000 7f88ee4d9640 10 monclient: monmap: 2024-03-29T11:31:30.525 INFO:teuthology.misc.health.smithi086.stderr:epoch 0 2024-03-29T11:31:30.525 INFO:teuthology.misc.health.smithi086.stderr:fsid 00000000-0000-0000-0000-000000000000 2024-03-29T11:31:30.525 INFO:teuthology.misc.health.smithi086.stderr:last_changed 0.000000 2024-03-29T11:31:30.525 INFO:teuthology.misc.health.smithi086.stderr:created 0.000000 2024-03-29T11:31:30.525 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 0 (unknown) 2024-03-29T11:31:30.525 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:31:30.525 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.noname-a 2024-03-29T11:31:30.525 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.noname-b 2024-03-29T11:31:30.525 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.noname-c 2024-03-29T11:31:30.525 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:31:30.526 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.522+0000 7f88ee4d9640 1 librados: starting msgr at 2024-03-29T11:31:30.526 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.522+0000 7f88ee4d9640 1 librados: starting objecter 2024-03-29T11:31:30.526 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.522+0000 7f88ee4d9640 1 Processor -- start 2024-03-29T11:31:30.526 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.522+0000 7f88ee4d9640 1 -- start start 2024-03-29T11:31:30.526 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.522+0000 7f88ee4d9640 1 librados: setting wanted keys 2024-03-29T11:31:30.526 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.522+0000 7f88ee4d9640 1 librados: calling monclient init 2024-03-29T11:31:30.526 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.522+0000 7f88ee4d9640 10 monclient: init 2024-03-29T11:31:30.526 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.523+0000 7f88ee4d9640 10 monclient: _reopen_session rank -1 2024-03-29T11:31:30.526 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.523+0000 7f88ee4d9640 10 monclient: _add_conns ranks=[0,1,2] 2024-03-29T11:31:30.526 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.523+0000 7f88ee4d9640 10 monclient(hunting): picked mon.noname-a con 0x7f88e80b17d0 addr v1:172.21.15.86:6789/0 2024-03-29T11:31:30.526 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.523+0000 7f88ee4d9640 10 monclient(hunting): picked mon.noname-b con 0x7f88e81422b0 addr v1:172.21.15.196:6789/0 2024-03-29T11:31:30.526 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.523+0000 7f88ee4d9640 10 monclient(hunting): picked mon.noname-c con 0x7f88e8142690 addr v1:172.21.15.86:6790/0 2024-03-29T11:31:30.526 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.523+0000 7f88ee4d9640 1 -- --> v1:172.21.15.86:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88e80accd0 con 0x7f88e80b17d0 2024-03-29T11:31:30.526 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.523+0000 7f88ee4d9640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88e80acf40 con 0x7f88e81422b0 2024-03-29T11:31:30.527 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.523+0000 7f88ee4d9640 1 -- --> v1:172.21.15.86:6790/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88e80ad220 con 0x7f88e8142690 2024-03-29T11:31:30.527 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.523+0000 7f88ee4d9640 10 monclient(hunting): _renew_subs 2024-03-29T11:31:30.527 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.523+0000 7f88ee4d9640 10 monclient(hunting): authenticate will time out at 1514.841125s 2024-03-29T11:31:30.527 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.523+0000 7f88e7fff640 1 --1- >> v1:172.21.15.86:6789/0 conn(0x7f88e80b17d0 0x7f88e81da990 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=1).handle_server_banner_and_identify peer v1:172.21.15.86:6789/0 says I am v1:172.21.15.86:36930/0 (socket says 172.21.15.86:36930) 2024-03-29T11:31:30.527 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.523+0000 7f88e7fff640 1 -- 172.21.15.86:0/15874257 learned_addr learned my addr 172.21.15.86:0/15874257 (peer_addr_for_me v1:172.21.15.86:0/0) 2024-03-29T11:31:30.527 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.523+0000 7f88e4ff9640 10 client.?.objecter ms_handle_connect 0x7f88e80b17d0 2024-03-29T11:31:30.527 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.523+0000 7f88e4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:30.527 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.524+0000 7f88e4ff9640 10 client.?.objecter ms_handle_connect 0x7f88e8142690 2024-03-29T11:31:30.527 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.524+0000 7f88e4ff9640 10 client.?.objecter resend_mon_ops 2024-03-29T11:31:30.528 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.524+0000 7f88e4ff9640 1 -- 172.21.15.86:0/15874257 <== mon.2 v1:172.21.15.86:6790/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4074439780 0 0) 0x7f88e80ad220 con 0x7f88e8142690 2024-03-29T11:31:30.528 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.524+0000 7f88e4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:31:30.528 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.524+0000 7f88e4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:30.528 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.525+0000 7f88e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:30.528 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.525+0000 7f88e4ff9640 10 monclient(hunting): my global_id is 4697 2024-03-29T11:31:30.528 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.525+0000 7f88e4ff9640 1 -- 172.21.15.86:0/15874257 --> v1:172.21.15.86:6790/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88d4004eb0 con 0x7f88e8142690 2024-03-29T11:31:30.528 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.525+0000 7f88e4ff9640 1 -- 172.21.15.86:0/15874257 <== mon.0 v1:172.21.15.86:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2323371835 0 0) 0x7f88e80accd0 con 0x7f88e80b17d0 2024-03-29T11:31:30.528 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.525+0000 7f88e4ff9640 10 monclient(hunting): _negotiate 2024-03-29T11:31:30.528 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.525+0000 7f88e4ff9640 10 monclient(hunting): _init_auth method 2 2024-03-29T11:31:30.528 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.525+0000 7f88e4ff9640 10 monclient(hunting): _init_auth creating new auth 2024-03-29T11:31:30.528 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.525+0000 7f88e4ff9640 10 monclient(hunting): my global_id is 4584 2024-03-29T11:31:30.529 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.525+0000 7f88e4ff9640 1 -- 172.21.15.86:0/15874257 --> v1:172.21.15.86:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88e80ad220 con 0x7f88e80b17d0 2024-03-29T11:31:30.529 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.525+0000 7f88e4ff9640 1 -- 172.21.15.86:0/15874257 <== mon.2 v1:172.21.15.86:6790/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 764+0+0 (unknown 1225979315 0 0) 0x7f88d4004eb0 con 0x7f88e8142690 2024-03-29T11:31:30.529 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.525+0000 7f88e4ff9640 1 -- 172.21.15.86:0/15874257 --> v1:172.21.15.86:6790/0 -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f88e80accd0 con 0x7f88e8142690 2024-03-29T11:31:30.529 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.525+0000 7f88e4ff9640 1 -- 172.21.15.86:0/15874257 <== mon.2 v1:172.21.15.86:6790/0 3 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f88e0001c30 con 0x7f88e8142690 2024-03-29T11:31:30.529 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.526+0000 7f88e4ff9640 10 monclient(hunting): handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:30.529 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.526+0000 7f88e4ff9640 10 monclient(hunting): got monmap 1 from mon.noname-c (according to old e1) 2024-03-29T11:31:30.529 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.526+0000 7f88e4ff9640 10 monclient(hunting): dump: 2024-03-29T11:31:30.529 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:31:30.529 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:31:30.529 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:30.529 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:30.529 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:30.529 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:31:30.529 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:31:30.529 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:31:30.530 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:31:30.530 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:31:30.530 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.526+0000 7f88e4ff9640 10 monclient(hunting): _finish_auth 0 2024-03-29T11:31:30.530 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.526+0000 7f88e4ff9640 1 -- 172.21.15.86:0/15874257 <== mon.2 v1:172.21.15.86:6790/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (unknown 3565965731 0 0) 0x7f88e80accd0 con 0x7f88e8142690 2024-03-29T11:31:30.530 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.526+0000 7f88e4ff9640 1 -- 172.21.15.86:0/15874257 >> v1:172.21.15.196:6789/0 conn(0x7f88e81422b0 legacy=0x7f88e80b0850 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-29T11:31:30.530 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.527+0000 7f88e4ff9640 1 -- 172.21.15.86:0/15874257 >> v1:172.21.15.86:6789/0 conn(0x7f88e80b17d0 legacy=0x7f88e81da990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:30.530 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.527+0000 7f88e4ff9640 10 monclient: _finish_hunting 0 2024-03-29T11:31:30.530 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.527+0000 7f88e4ff9640 1 monclient: found mon.c 2024-03-29T11:31:30.530 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.527+0000 7f88e4ff9640 20 monclient: _un_backoff reopen_interval_multipler now 1 2024-03-29T11:31:30.530 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.527+0000 7f88e4ff9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:31:30.530 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.527+0000 7f88e4ff9640 1 -- 172.21.15.86:0/15874257 --> v1:172.21.15.86:6790/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88e813e550 con 0x7f88e8142690 2024-03-29T11:31:30.530 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.527+0000 7f88e4ff9640 10 monclient: _finish_auth 0 2024-03-29T11:31:30.531 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.527+0000 7f88e4ff9640 10 monclient: _check_auth_tickets 2024-03-29T11:31:30.531 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.527+0000 7f88e4ff9640 20 monclient: _check_auth_rotating not needed by client.admin 2024-03-29T11:31:30.531 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.527+0000 7f88ee4d9640 5 monclient: authenticate success, global_id 4697 2024-03-29T11:31:30.531 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.527+0000 7f88ee4d9640 10 monclient: _renew_subs 2024-03-29T11:31:30.531 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.527+0000 7f88ee4d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:31:30.531 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.527+0000 7f88ee4d9640 1 -- 172.21.15.86:0/15874257 --> v1:172.21.15.86:6790/0 -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88e81db0a0 con 0x7f88e8142690 2024-03-29T11:31:30.531 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.527+0000 7f88ee4d9640 10 client.4697.objecter _maybe_request_map subscribing (onetime) to next osd map 2024-03-29T11:31:30.531 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.527+0000 7f88ee4d9640 10 monclient: _renew_subs 2024-03-29T11:31:30.531 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.527+0000 7f88ee4d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:31:30.531 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.527+0000 7f88ee4d9640 1 -- 172.21.15.86:0/15874257 --> v1:172.21.15.86:6790/0 -- mon_subscribe({osdmap=0}) v3 -- 0x7f88e8150f30 con 0x7f88e8142690 2024-03-29T11:31:30.531 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.527+0000 7f88ee4d9640 1 librados: init done 2024-03-29T11:31:30.531 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.527+0000 7f88e4ff9640 1 -- 172.21.15.86:0/15874257 <== mon.2 v1:172.21.15.86:6790/0 5 ==== config(0 keys) v1 ==== 4+0+0 (unknown 0 0 0) 0x7f88e0003990 con 0x7f88e8142690 2024-03-29T11:31:30.531 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.527+0000 7f88e4ff9640 10 monclient: handle_config config(0 keys) v1 2024-03-29T11:31:30.531 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.527+0000 7f88e4ff9640 1 -- 172.21.15.86:0/15874257 <== mon.2 v1:172.21.15.86:6790/0 6 ==== mon_map magic: 0 v1 ==== 308+0+0 (unknown 2354954938 0 0) 0x7f88e0003ca0 con 0x7f88e8142690 2024-03-29T11:31:30.532 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.527+0000 7f88e4ff9640 10 monclient: handle_monmap mon_map magic: 0 v1 2024-03-29T11:31:30.532 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.527+0000 7f88e4ff9640 10 monclient: got monmap 1 from mon.c (according to old e1) 2024-03-29T11:31:30.532 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.527+0000 7f88e4ff9640 10 monclient: dump: 2024-03-29T11:31:30.532 INFO:teuthology.misc.health.smithi086.stderr:epoch 1 2024-03-29T11:31:30.532 INFO:teuthology.misc.health.smithi086.stderr:fsid 12b413e9-4891-4542-8b6d-70acb06d2a3d 2024-03-29T11:31:30.532 INFO:teuthology.misc.health.smithi086.stderr:last_changed 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:30.532 INFO:teuthology.misc.health.smithi086.stderr:created 2024-03-29T11:19:55.301352+0000 2024-03-29T11:31:30.532 INFO:teuthology.misc.health.smithi086.stderr:min_mon_release 19 (squid) 2024-03-29T11:31:30.532 INFO:teuthology.misc.health.smithi086.stderr:election_strategy: 1 2024-03-29T11:31:30.532 INFO:teuthology.misc.health.smithi086.stderr:0: v1:172.21.15.86:6789/0 mon.a 2024-03-29T11:31:30.532 INFO:teuthology.misc.health.smithi086.stderr:1: v1:172.21.15.196:6789/0 mon.b 2024-03-29T11:31:30.532 INFO:teuthology.misc.health.smithi086.stderr:2: v1:172.21.15.86:6790/0 mon.c 2024-03-29T11:31:30.532 INFO:teuthology.misc.health.smithi086.stderr: 2024-03-29T11:31:30.533 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.528+0000 7f88e5ffb640 4 set_mon_vals no callback set 2024-03-29T11:31:30.533 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.529+0000 7f88ee4d9640 10 start_mon_command cmd=[{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:30.533 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.529+0000 7f88ee4d9640 10 monclient: _send_command 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:30.533 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.529+0000 7f88ee4d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:31:30.533 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.529+0000 7f88ee4d9640 1 -- 172.21.15.86:0/15874257 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88c8005080 con 0x7f88e8142690 2024-03-29T11:31:30.534 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.529+0000 7f88e4ff9640 1 -- 172.21.15.86:0/15874257 <== mon.2 v1:172.21.15.86:6790/0 7 ==== mgrmap(e 4) v1 ==== 89697+0+0 (unknown 2570754144 0 0) 0x7f88e0019c70 con 0x7f88e8142690 2024-03-29T11:31:30.534 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.530+0000 7f88e4ff9640 1 -- 172.21.15.86:0/15874257 <== mon.2 v1:172.21.15.86:6790/0 8 ==== osd_map(11..11 src has 1..11) v4 ==== 1645+0+0 (unknown 2923349858 0 0) 0x7f88e0083a90 con 0x7f88e8142690 2024-03-29T11:31:30.534 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.530+0000 7f88e4ff9640 10 client.4697.objecter ms_dispatch 0x7f88e8000b90 osd_map(11..11 src has 1..11) v4 2024-03-29T11:31:30.534 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.530+0000 7f88e4ff9640 3 client.4697.objecter handle_osd_map got epochs [11,11] > 0 2024-03-29T11:31:30.534 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.530+0000 7f88e4ff9640 3 client.4697.objecter handle_osd_map decoding full epoch 11 2024-03-29T11:31:30.534 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.530+0000 7f88e4ff9640 20 client.4697.objecter dump_active .. 0 homeless 2024-03-29T11:31:30.535 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.530+0000 7f88e7fff640 10 monclient: get_auth_request con 0x7f88d406db60 auth_method 0 2024-03-29T11:31:30.535 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.531+0000 7f88e4ff9640 10 client.4697.objecter ms_handle_connect 0x7f88d406db60 2024-03-29T11:31:30.535 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.537+0000 7f88e4ff9640 1 -- 172.21.15.86:0/15874257 <== mon.2 v1:172.21.15.86:6790/0 9 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+182440 (unknown 1092875540 0 2680962430) 0x7f88e004f370 con 0x7f88e8142690 2024-03-29T11:31:30.535 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.538+0000 7f88e4ff9640 10 monclient: handle_mon_command_ack 1 [{"prefix": "get_command_descriptions"}] 2024-03-29T11:31:30.535 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.538+0000 7f88e4ff9640 10 monclient: _finish_command 1 = system:0 2024-03-29T11:31:30.850 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.856+0000 7f88ee4d9640 10 start_mon_command cmd=[{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:30.851 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.856+0000 7f88ee4d9640 10 monclient: _send_command 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:30.851 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.856+0000 7f88ee4d9640 10 monclient: _send_mon_message to mon.c at v1:172.21.15.86:6790/0 2024-03-29T11:31:30.851 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.856+0000 7f88ee4d9640 1 -- 172.21.15.86:0/15874257 --> v1:172.21.15.86:6790/0 -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f88c8005330 con 0x7f88e8142690 2024-03-29T11:31:30.851 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.857+0000 7f88e4ff9640 1 -- 172.21.15.86:0/15874257 <== mon.2 v1:172.21.15.86:6790/0 10 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+7100 (unknown 238102980 0 3394516648) 0x7f88e00512a0 con 0x7f88e8142690 2024-03-29T11:31:30.851 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.857+0000 7f88e4ff9640 10 monclient: handle_mon_command_ack 2 [{"prefix": "osd dump", "format": "json"}] 2024-03-29T11:31:30.852 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.857+0000 7f88e4ff9640 10 monclient: _finish_command 2 = system:0 2024-03-29T11:31:30.855 INFO:teuthology.misc.health.smithi086.stdout: 2024-03-29T11:31:30.855 INFO:teuthology.misc.health.smithi086.stdout:{"epoch":11,"fsid":"12b413e9-4891-4542-8b6d-70acb06d2a3d","created":"2024-03-29T11:20:33.510001+0000","modified":"2024-03-29T11:30:38.448721+0000","last_up_change":"0.000000","last_in_change":"2024-03-29T11:30:38.448721+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"69e25ec4-1959-46d9-8d55-eb4e52eaf6f1","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":1,"uuid":"8378e0ad-8983-46ba-a2f1-24b7fc3c7df4","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":2,"uuid":"a5d7c244-1f0f-4632-8246-1226c2c180d5","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["autoout","exists","new"]},{"osd":3,"uuid":"a4113954-1342-43e6-8807-bddf6ae9d460","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"04a3b66e-d800-443e-ade7-184a154ee5a3","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"3ed0352b-96f6-4668-88dd-4747f5385eb4","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"ae0370e8-7641-4e0b-9d5d-e2d8a6175a17","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"3c1de7fe-970f-4063-a391-bb16258fb567","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":65536,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-29T11:31:30.856 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.861+0000 7f88ee4d9640 10 librados: watch_flush enter 2024-03-29T11:31:30.856 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.862+0000 7f88ee4d9640 10 librados: watch_flush exit 2024-03-29T11:31:30.856 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.862+0000 7f88ee4d9640 20 client.4697.objecter shutdown clearing up homeless session... 2024-03-29T11:31:30.857 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.862+0000 7f88ee4d9640 10 client.4697.objecter successfully canceled tick 2024-03-29T11:31:30.857 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.862+0000 7f88ee4d9640 1 -- 172.21.15.86:0/15874257 >> v1:172.21.15.196:6816/588110318 conn(0x7f88d406db60 legacy=0x7f88d4070020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:30.857 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.862+0000 7f88ee4d9640 10 monclient: shutdown 2024-03-29T11:31:30.857 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.862+0000 7f88ee4d9640 20 monclient: shutdown discarding 0 pending message(s) 2024-03-29T11:31:30.857 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.862+0000 7f88ee4d9640 1 -- 172.21.15.86:0/15874257 >> v1:172.21.15.86:6790/0 conn(0x7f88e8142690 legacy=0x7f88e81e2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-29T11:31:30.857 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.862+0000 7f88ee4d9640 1 -- 172.21.15.86:0/15874257 shutdown_connections 2024-03-29T11:31:30.857 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.862+0000 7f88ee4d9640 1 -- 172.21.15.86:0/15874257 >> 172.21.15.86:0/15874257 conn(0x7f88e8066a00 msgr2=0x7f88e813c2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-29T11:31:30.858 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.864+0000 7f88ee4d9640 1 -- 172.21.15.86:0/15874257 shutdown_connections 2024-03-29T11:31:30.858 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.864+0000 7f88ee4d9640 1 -- 172.21.15.86:0/15874257 wait complete. 2024-03-29T11:31:30.859 INFO:teuthology.misc.health.smithi086.stderr:2024-03-29T11:31:30.865+0000 7f88ee4d9640 1 librados: shutdown 2024-03-29T11:31:30.881 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-03-29T11:31:30.881 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:31:30.883 INFO:teuthology.misc:Shutting down osd daemons... 2024-03-29T11:31:30.883 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-03-29T11:31:30.883 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:31:30.921 INFO:tasks.ceph.osd.0:Stopped 2024-03-29T11:31:30.921 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-03-29T11:31:30.922 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:31:30.991 INFO:tasks.ceph.osd.1:Stopped 2024-03-29T11:31:30.991 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-03-29T11:31:30.991 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:31:31.067 INFO:tasks.ceph.osd.2:Stopped 2024-03-29T11:31:31.067 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-03-29T11:31:31.067 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:31:31.137 INFO:tasks.ceph.osd.3:Stopped 2024-03-29T11:31:31.138 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-03-29T11:31:31.138 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:31:31.178 INFO:tasks.ceph.osd.4:Stopped 2024-03-29T11:31:31.179 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-03-29T11:31:31.179 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:31:31.255 INFO:tasks.ceph.osd.5:Stopped 2024-03-29T11:31:31.255 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-03-29T11:31:31.255 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:31:31.324 INFO:tasks.ceph.osd.6:Stopped 2024-03-29T11:31:31.324 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-03-29T11:31:31.324 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:31:31.393 INFO:tasks.ceph.osd.7:Stopped 2024-03-29T11:31:31.393 INFO:teuthology.misc:Shutting down mgr daemons... 2024-03-29T11:31:31.394 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-03-29T11:31:31.394 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:31:31.417 INFO:tasks.ceph.mgr.y:Stopped 2024-03-29T11:31:31.417 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-03-29T11:31:31.417 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:31:31.457 INFO:tasks.ceph.mgr.x:Stopped 2024-03-29T11:31:31.458 INFO:teuthology.misc:Shutting down mon daemons... 2024-03-29T11:31:31.458 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-03-29T11:31:31.458 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:31:31.470 INFO:tasks.ceph.mon.a:Stopped 2024-03-29T11:31:31.470 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-03-29T11:31:31.470 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:31:31.521 INFO:tasks.ceph.mon.c:Stopped 2024-03-29T11:31:31.521 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-03-29T11:31:31.521 INFO:teuthology.orchestra.run:waiting for 300 2024-03-29T11:31:31.532 INFO:tasks.ceph.mon.b:Stopped 2024-03-29T11:31:31.532 INFO:tasks.ceph:Checking cluster log for badness... 2024-03-29T11:31:31.533 DEBUG:teuthology.orchestra.run.smithi086:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | egrep -v 'reached quota' | egrep -v 'overall HEALTH_' | egrep -v '\(CACHE_POOL_NO_HIT_SET\)' | egrep -v '\(POOL_FULL\)' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(CACHE_POOL_NEAR_FULL\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v '\(PG_AVAILABILITY\)' | egrep -v '\(OBJECT_MISPLACED\)' | egrep -v 'slow request' | head -n 1 2024-03-29T11:31:31.647 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi086.front.sepia.ceph.com 2024-03-29T11:31:31.647 DEBUG:teuthology.orchestra.run.smithi086:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-03-29T11:31:32.868 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi086.front.sepia.ceph.com 2024-03-29T11:31:32.868 DEBUG:teuthology.orchestra.run.smithi086:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-03-29T11:31:32.965 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi086.front.sepia.ceph.com 2024-03-29T11:31:32.965 DEBUG:teuthology.orchestra.run.smithi086:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-03-29T11:31:33.031 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi086.front.sepia.ceph.com 2024-03-29T11:31:33.032 DEBUG:teuthology.orchestra.run.smithi086:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-03-29T11:31:33.106 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi196.front.sepia.ceph.com 2024-03-29T11:31:33.107 DEBUG:teuthology.orchestra.run.smithi196:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-03-29T11:31:36.105 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi196.front.sepia.ceph.com 2024-03-29T11:31:36.105 DEBUG:teuthology.orchestra.run.smithi196:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-03-29T11:31:36.181 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi196.front.sepia.ceph.com 2024-03-29T11:31:36.181 DEBUG:teuthology.orchestra.run.smithi196:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-03-29T11:31:36.239 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi196.front.sepia.ceph.com 2024-03-29T11:31:36.239 DEBUG:teuthology.orchestra.run.smithi196:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-03-29T11:31:36.298 INFO:tasks.ceph:Archiving mon data... 2024-03-29T11:31:36.300 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622694/data/mon.a.tgz 2024-03-29T11:31:36.300 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2024-03-29T11:31:36.318 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.uYh5SrSQew 2024-03-29T11:31:36.319 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.uYh5SrSQew 2024-03-29T11:31:36.439 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.uYh5SrSQew 2024-03-29T11:31:36.526 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.uYh5SrSQew is 211KB 2024-03-29T11:31:36.620 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.uYh5SrSQew 2024-03-29T11:31:36.635 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622694/data/mon.c.tgz 2024-03-29T11:31:36.635 DEBUG:teuthology.orchestra.run.smithi086:> mktemp 2024-03-29T11:31:36.689 INFO:teuthology.orchestra.run.smithi086.stdout:/tmp/tmp.UyuugXoIV1 2024-03-29T11:31:36.689 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.UyuugXoIV1 2024-03-29T11:31:36.789 DEBUG:teuthology.orchestra.run.smithi086:> sudo chmod 0666 /tmp/tmp.UyuugXoIV1 2024-03-29T11:31:36.873 DEBUG:teuthology.orchestra.remote:smithi086:/tmp/tmp.UyuugXoIV1 is 211KB 2024-03-29T11:31:36.966 DEBUG:teuthology.orchestra.run.smithi086:> rm -fr /tmp/tmp.UyuugXoIV1 2024-03-29T11:31:36.981 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622694/data/mon.b.tgz 2024-03-29T11:31:36.981 DEBUG:teuthology.orchestra.run.smithi196:> mktemp 2024-03-29T11:31:36.999 INFO:teuthology.orchestra.run.smithi196.stdout:/tmp/tmp.JBQDyGnmd4 2024-03-29T11:31:36.999 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.JBQDyGnmd4 2024-03-29T11:31:37.113 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0666 /tmp/tmp.JBQDyGnmd4 2024-03-29T11:31:37.201 DEBUG:teuthology.orchestra.remote:smithi196:/tmp/tmp.JBQDyGnmd4 is 211KB 2024-03-29T11:31:37.296 DEBUG:teuthology.orchestra.run.smithi196:> rm -fr /tmp/tmp.JBQDyGnmd4 2024-03-29T11:31:37.312 INFO:tasks.ceph:Cleaning ceph cluster... 2024-03-29T11:31:37.312 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:31:37.314 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-03-29T11:31:37.397 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:31:37.442 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-03-29T11:31:37.443 INFO:tasks.ceph:Archiving crash dumps... 2024-03-29T11:31:37.930 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622694/remote/smithi086/crash 2024-03-29T11:31:37.941 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:31:37.973 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/lib/ceph/crash to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622694/remote/smithi196/crash 2024-03-29T11:31:37.973 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-03-29T11:31:38.002 INFO:tasks.ceph:Compressing logs... 2024-03-29T11:31:38.002 DEBUG:teuthology.orchestra.run.smithi086:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:31:38.017 DEBUG:teuthology.orchestra.run.smithi196:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-29T11:31:38.066 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38329.log 2024-03-29T11:31:38.066 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-03-29T11:31:38.066 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38329.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38368.log 2024-03-29T11:31:38.067 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40127.log 2024-03-29T11:31:38.067 INFO:teuthology.orchestra.run.smithi086.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38329.log.gz 2024-03-29T11:31:38.067 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38368.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38212.log 2024-03-29T11:31:38.067 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38914.log 2024-03-29T11:31:38.067 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40127.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38368.log.gz 2024-03-29T11:31:38.067 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38212.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40205.log 2024-03-29T11:31:38.067 INFO:teuthology.orchestra.run.smithi086.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40127.log.gz 2024-03-29T11:31:38.067 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38914.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38212.log.gz 2024-03-29T11:31:38.067 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39815.log 2024-03-29T11:31:38.067 INFO:teuthology.orchestra.run.smithi086.stderr: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38914.log.gz 2024-03-29T11:31:38.067 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40205.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39893.log 2024-03-29T11:31:38.067 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40751.log 2024-03-29T11:31:38.067 INFO:teuthology.orchestra.run.smithi086.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40205.log.gz 2024-03-29T11:31:38.068 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39893.log: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39815.log.gz 2024-03-29T11:31:38.068 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41339.log 2024-03-29T11:31:38.068 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40751.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.39893.log.gz 2024-03-29T11:31:38.068 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38758.log 2024-03-29T11:31:38.068 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41339.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41222.log 2024-03-29T11:31:38.068 INFO:teuthology.orchestra.run.smithi086.stderr: 85.3% -- replaced with /var/log/ceph/ceph-client.admin.40751.log.gz 2024-03-29T11:31:38.068 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38758.log: gzip -5 --verbose 85.8% -- -- replaced with /var/log/ceph/ceph-client.admin.41339.log.gz /var/log/ceph/ceph-client.admin.39616.log 2024-03-29T11:31:38.068 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T11:31:38.068 INFO:teuthology.orchestra.run.smithi086.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38758.log.gz 2024-03-29T11:31:38.068 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38524.log 2024-03-29T11:31:38.068 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40400.log 2024-03-29T11:31:38.068 INFO:teuthology.orchestra.run.smithi086.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41222.log.gz 2024-03-29T11:31:38.068 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38524.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39265.log 2024-03-29T11:31:38.068 INFO:teuthology.orchestra.run.smithi086.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.39616.log.gz 2024-03-29T11:31:38.068 INFO:teuthology.orchestra.run.smithi086.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38524.log.gz 2024-03-29T11:31:38.069 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40283.log 2024-03-29T11:31:38.069 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39265.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40400.log.gz 2024-03-29T11:31:38.069 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39343.log 2024-03-29T11:31:38.069 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40283.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39265.log.gz 2024-03-29T11:31:38.069 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41105.log 2024-03-29T11:31:38.069 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39343.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40283.log.gz 2024-03-29T11:31:38.069 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40049.log 2024-03-29T11:31:38.069 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40478.log 2024-03-29T11:31:38.069 INFO:teuthology.orchestra.run.smithi086.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39343.log.gz 2024-03-29T11:31:38.069 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40049.log: gzip -5 --verbose -- 86.7% /var/log/ceph/ceph-client.admin.39226.log 2024-03-29T11:31:38.069 INFO:teuthology.orchestra.run.smithi086.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41105.log.gz 2024-03-29T11:31:38.069 INFO:teuthology.orchestra.run.smithi086.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.40049.log.gz 2024-03-29T11:31:38.069 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40478.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41066.log 2024-03-29T11:31:38.069 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39226.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39070.log 2024-03-29T11:31:38.070 INFO:teuthology.orchestra.run.smithi086.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.40478.log.gz 2024-03-29T11:31:38.070 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41066.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39226.log.gz 2024-03-29T11:31:38.070 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38875.log 2024-03-29T11:31:38.070 INFO:teuthology.orchestra.run.smithi086.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.41066.log.gz 2024-03-29T11:31:38.070 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37937.log 2024-03-29T11:31:38.070 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40712.log 2024-03-29T11:31:38.070 INFO:teuthology.orchestra.run.smithi086.stderr: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.39070.log.gz 2024-03-29T11:31:38.070 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37937.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38875.log.gz 2024-03-29T11:31:38.070 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:31:38.070 INFO:teuthology.orchestra.run.smithi086.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.37937.log.gz 2024-03-29T11:31:38.070 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40712.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39655.log 2024-03-29T11:31:38.070 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39932.log 2024-03-29T11:31:38.070 INFO:teuthology.orchestra.run.smithi086.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.40712.log.gz 2024-03-29T11:31:38.070 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39655.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40829.log 2024-03-29T11:31:38.071 INFO:teuthology.orchestra.run.smithi086.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39655.log.gz 2024-03-29T11:31:38.071 INFO:teuthology.orchestra.run.smithi086.stderr: 93.5% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:31:38.071 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39932.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39109.log 2024-03-29T11:31:38.071 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40829.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39932.log.gz 2024-03-29T11:31:38.071 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38173.log 2024-03-29T11:31:38.071 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39109.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40829.log.gz 2024-03-29T11:31:38.071 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38719.log 2024-03-29T11:31:38.071 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38173.log: 86.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.39109.log.gz 2024-03-29T11:31:38.071 INFO:teuthology.orchestra.run.smithi086.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40439.log 2024-03-29T11:31:38.071 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38719.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39854.log 2024-03-29T11:31:38.072 INFO:teuthology.orchestra.run.smithi086.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38173.log.gz 2024-03-29T11:31:38.072 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40439.log: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.38719.log.gz 2024-03-29T11:31:38.072 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38407.log 2024-03-29T11:31:38.072 INFO:teuthology.orchestra.run.smithi086.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.40439.log.gz 2024-03-29T11:31:38.072 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39854.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40010.log 2024-03-29T11:31:38.072 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38992.log 2024-03-29T11:31:38.072 INFO:teuthology.orchestra.run.smithi086.stderr: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.39854.log.gz 2024-03-29T11:31:38.072 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40010.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38407.log.gz 2024-03-29T11:31:38.072 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-03-29T11:31:38.072 INFO:teuthology.orchestra.run.smithi086.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.40010.log.gz 2024-03-29T11:31:38.072 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38992.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41261.log 2024-03-29T11:31:38.072 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38093.log 2024-03-29T11:31:38.072 INFO:teuthology.orchestra.run.smithi086.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38992.log.gz 2024-03-29T11:31:38.072 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41261.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38680.log 2024-03-29T11:31:38.073 INFO:teuthology.orchestra.run.smithi086.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.41261.log.gz 2024-03-29T11:31:38.073 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38093.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40361.log 2024-03-29T11:31:38.073 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37898.log 2024-03-29T11:31:38.073 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38680.log: 87.1% -- replaced with /var/log/ceph/ceph-client.admin.38093.log.gz 2024-03-29T11:31:38.073 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40361.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38446.log 2024-03-29T11:31:38.073 INFO:teuthology.orchestra.run.smithi086.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38680.log.gz 2024-03-29T11:31:38.073 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37898.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40361.log.gz 2024-03-29T11:31:38.073 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40790.log 2024-03-29T11:31:38.073 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38446.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.37898.log.gz 2024-03-29T11:31:38.073 INFO:teuthology.orchestra.run.smithi086.stderr: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.38446.log.gz 2024-03-29T11:31:38.073 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39538.log 2024-03-29T11:31:38.073 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40790.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39421.log 2024-03-29T11:31:38.073 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39538.log: gzip -5 86.3% --verbose -- -- replaced with /var/log/ceph/ceph-client.admin.40790.log.gz /var/log/ceph/ceph-client.admin.39776.log 2024-03-29T11:31:38.073 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T11:31:38.073 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39421.log: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.39538.log.gz 2024-03-29T11:31:38.074 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34286.log 2024-03-29T11:31:38.074 INFO:teuthology.orchestra.run.smithi086.stderr: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.39421.log.gz 2024-03-29T11:31:38.074 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39499.log 2024-03-29T11:31:38.074 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39776.log: /var/log/ceph/ceph-client.admin.34286.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40868.log 2024-03-29T11:31:38.074 INFO:teuthology.orchestra.run.smithi086.stderr: 85.9% -- replaced with /var/log/ceph/ceph-client.admin.39776.log.gz 2024-03-29T11:31:38.074 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39499.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38641.log 2024-03-29T11:31:38.074 INFO:teuthology.orchestra.run.smithi086.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.34286.log.gz 2024-03-29T11:31:38.074 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40868.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38290.log 2024-03-29T11:31:38.074 INFO:teuthology.orchestra.run.smithi086.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39499.log.gz 2024-03-29T11:31:38.074 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38641.log: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.40868.log.gz 2024-03-29T11:31:38.074 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40517.log 2024-03-29T11:31:38.074 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38953.log 2024-03-29T11:31:38.074 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38290.log: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.38641.log.gz 2024-03-29T11:31:38.074 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40517.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-03-29T11:31:38.075 INFO:teuthology.orchestra.run.smithi086.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38290.log.gz 2024-03-29T11:31:38.075 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38953.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40517.log.gz 2024-03-29T11:31:38.075 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41144.log 2024-03-29T11:31:38.075 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-osd.3.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38953.log.gz 2024-03-29T11:31:38.075 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38485.log 2024-03-29T11:31:38.090 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-03-29T11:31:38.090 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34458.log 2024-03-29T11:31:38.090 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34618.log 2024-03-29T11:31:38.090 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-client.admin.34458.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34658.log 2024-03-29T11:31:38.090 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.34618.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-03-29T11:31:38.091 INFO:teuthology.orchestra.run.smithi196.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34458.log.gz 2024-03-29T11:31:38.091 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-03-29T11:31:38.091 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.34658.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34618.log.gz 2024-03-29T11:31:38.091 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-03-29T11:31:38.091 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-03-29T11:31:38.091 INFO:teuthology.orchestra.run.smithi196.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34658.log.gz 2024-03-29T11:31:38.091 INFO:teuthology.orchestra.run.smithi196.stderr: 88.2% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-03-29T11:31:38.091 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34418.log 2024-03-29T11:31:38.091 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-03-29T11:31:38.091 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.34418.log: gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:31:38.091 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34378.log 88.0% -- replaced with /var/log/ceph/ceph-client.admin.34418.log.gz 2024-03-29T11:31:38.091 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:31:38.091 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34538.log 2024-03-29T11:31:38.092 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.34378.log: gzip -5 --verbose -- 87.2% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:31:38.092 INFO:teuthology.orchestra.run.smithi196.stderr: /var/log/ceph/ceph-client.admin.34498.log 2024-03-29T11:31:38.092 INFO:teuthology.orchestra.run.smithi196.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34378.log.gz 2024-03-29T11:31:38.092 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-03-29T11:31:38.092 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.34538.log: /var/log/ceph/ceph-client.admin.34498.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34578.log 2024-03-29T11:31:38.092 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-mon.b.log: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34498.log.gz 2024-03-29T11:31:38.092 INFO:teuthology.orchestra.run.smithi196.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.34538.log.gz 2024-03-29T11:31:38.092 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.34578.log: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.34578.log.gz 2024-03-29T11:31:38.107 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40322.log 2024-03-29T11:31:38.107 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41144.log: /var/log/ceph/ceph-client.admin.38485.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39148.log 2024-03-29T11:31:38.108 INFO:teuthology.orchestra.run.smithi086.stderr: 86.8% -- replaced with /var/log/ceph/ceph-client.admin.41144.log.gz 2024-03-29T11:31:38.108 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40322.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40088.log 2024-03-29T11:31:38.108 INFO:teuthology.orchestra.run.smithi086.stderr: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.38485.log.gz 2024-03-29T11:31:38.108 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39148.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.40322.log.gz 2024-03-29T11:31:38.108 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34680.log 2024-03-29T11:31:38.108 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40088.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39460.log 2024-03-29T11:31:38.108 INFO:teuthology.orchestra.run.smithi086.stderr: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39148.log.gz 2024-03-29T11:31:38.108 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34680.log: gzip -5 --verbose -- 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40088.log.gz 2024-03-29T11:31:38.108 INFO:teuthology.orchestra.run.smithi086.stderr: /var/log/ceph/ceph-client.admin.39382.log 2024-03-29T11:31:38.108 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39460.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.34680.log.gz 2024-03-29T11:31:38.108 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40988.log 2024-03-29T11:31:38.108 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41027.log 2024-03-29T11:31:38.109 INFO:teuthology.orchestra.run.smithi086.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.39460.log.gz 2024-03-29T11:31:38.109 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39382.log: /var/log/ceph/ceph-client.admin.40988.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40673.log 2024-03-29T11:31:38.109 INFO:teuthology.orchestra.run.smithi086.stderr: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39382.log.gz 2024-03-29T11:31:38.109 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41027.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.40988.log.gz 2024-03-29T11:31:38.109 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40556.log 2024-03-29T11:31:38.109 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40910.log 2024-03-29T11:31:38.109 INFO:teuthology.orchestra.run.smithi086.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.41027.log.gz 2024-03-29T11:31:38.109 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40556.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.y.log 2024-03-29T11:31:38.109 INFO:teuthology.orchestra.run.smithi086.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.40673.log.gz 2024-03-29T11:31:38.109 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40910.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41183.log 2024-03-29T11:31:38.109 INFO:teuthology.orchestra.run.smithi086.stderr: 86.1% -- replaced with /var/log/ceph/ceph-client.admin.40556.log.gz 2024-03-29T11:31:38.109 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-mgr.y.log: 85.5% -- replaced with /var/log/ceph/ceph-client.admin.40910.log.gz 2024-03-29T11:31:38.109 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.28478.log 2024-03-29T11:31:38.110 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40949.log 2024-03-29T11:31:38.110 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41183.log: /var/log/ceph/ceph.tmp-client.admin.28478.log: gzip -5 --verbose -- 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.28478.log.gz 2024-03-29T11:31:38.110 INFO:teuthology.orchestra.run.smithi086.stderr: /var/log/ceph/ceph-client.admin.38563.log 2024-03-29T11:31:38.110 INFO:teuthology.orchestra.run.smithi086.stderr: 87.3% -- replaced with /var/log/ceph/ceph-client.admin.41183.log.gz 2024-03-29T11:31:38.110 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40949.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40634.log 2024-03-29T11:31:38.110 INFO:teuthology.orchestra.run.smithi086.stderr: 85.6% -- replaced with /var/log/ceph/ceph-client.admin.40949.log.gz 2024-03-29T11:31:38.110 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38563.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-03-29T11:31:38.110 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40634.log: 86.9% -- replaced with /var/log/ceph/ceph-client.admin.38563.log.gz 2024-03-29T11:31:38.110 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39187.log 2024-03-29T11:31:38.110 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-osd.0.log: gzip 87.2% -- replaced with /var/log/ceph/ceph-client.admin.40634.log.gz 2024-03-29T11:31:38.110 INFO:teuthology.orchestra.run.smithi086.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.40244.log 2024-03-29T11:31:38.110 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39187.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40166.log 2024-03-29T11:31:38.110 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40244.log: 85.7% -- replaced with /var/log/ceph/ceph-client.admin.39187.log.gz 2024-03-29T11:31:38.110 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-03-29T11:31:38.111 INFO:teuthology.orchestra.run.smithi086.stderr: 86.6%gzip -- replaced with /var/log/ceph/ceph-client.admin.40244.log.gz -5 --verbose -- /var/log/ceph/ceph-client.admin.38132.log 2024-03-29T11:31:38.111 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T11:31:38.111 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40166.log: /var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38602.log 2024-03-29T11:31:38.111 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38132.log: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.40166.log.gz 2024-03-29T11:31:38.111 INFO:teuthology.orchestra.run.smithi086.stderr: 86.0% -- replaced with /var/log/ceph/ceph-client.admin.38132.log.gzgzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-03-29T11:31:38.111 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T11:31:38.111 INFO:teuthology.orchestra.run.smithi086.stderr: 92.7% -- replaced with /var/log/ceph/ceph.log.gz 2024-03-29T11:31:38.111 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38836.log 2024-03-29T11:31:38.111 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38602.log: /var/log/ceph/ceph-osd.2.log: 86.6% -- replaced with /var/log/ceph/ceph-client.admin.38602.log.gz 2024-03-29T11:31:38.111 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39577.log 2024-03-29T11:31:38.111 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38836.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40595.log 2024-03-29T11:31:38.111 INFO:teuthology.orchestra.run.smithi086.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38836.log.gz 2024-03-29T11:31:38.111 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39577.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.39577.log.gz 2024-03-29T11:31:38.111 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38015.log 2024-03-29T11:31:38.111 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39697.log 2024-03-29T11:31:38.111 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.40595.log: /var/log/ceph/ceph-client.admin.38015.log: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.40595.log.gz 2024-03-29T11:31:38.112 INFO:teuthology.orchestra.run.smithi086.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.38015.log.gz 2024-03-29T11:31:38.112 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39031.log 2024-03-29T11:31:38.112 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39697.log: 93.8% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-03-29T11:31:38.112 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39737.log 2024-03-29T11:31:38.112 INFO:teuthology.orchestra.run.smithi086.stderr: 86.5% -- replaced with /var/log/ceph/ceph-client.admin.39697.log.gz 2024-03-29T11:31:38.112 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41300.log 2024-03-29T11:31:38.112 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39737.log: 86.9%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38054.log 2024-03-29T11:31:38.112 INFO:teuthology.orchestra.run.smithi086.stderr: -- replaced with /var/log/ceph/ceph-client.admin.39031.log.gz 2024-03-29T11:31:38.112 INFO:teuthology.orchestra.run.smithi086.stderr:gzip 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39737.log.gz 2024-03-29T11:31:38.112 INFO:teuthology.orchestra.run.smithi086.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.38797.log 2024-03-29T11:31:38.112 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.41300.log: /var/log/ceph/ceph-client.admin.38054.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.34617.log 2024-03-29T11:31:38.112 INFO:teuthology.orchestra.run.smithi086.stderr: 86.3% 85.7% -- replaced with /var/log/ceph/ceph-client.admin.41300.log.gz 2024-03-29T11:31:38.112 INFO:teuthology.orchestra.run.smithi086.stderr: -- replaced with /var/log/ceph/ceph-client.admin.38054.log.gz 2024-03-29T11:31:38.113 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38797.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.38251.log 2024-03-29T11:31:38.113 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.34617.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39971.log 2024-03-29T11:31:38.113 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.38251.log: 86.4% -- replaced with /var/log/ceph/ceph-client.admin.38797.log.gz 2024-03-29T11:31:38.113 INFO:teuthology.orchestra.run.smithi086.stderr: 87.0% -- replaced with /var/log/ceph/ceph-client.admin.34617.log.gz 2024-03-29T11:31:38.113 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39304.log 2024-03-29T11:31:38.113 INFO:teuthology.orchestra.run.smithi086.stderr: 86.3% -- replaced with /var/log/ceph/ceph-client.admin.38251.log.gz 2024-03-29T11:31:38.113 INFO:teuthology.orchestra.run.smithi086.stderr:gzip -5/var/log/ceph/ceph-client.admin.39971.log: --verbose -- /var/log/ceph/ceph-mon.c.log 2024-03-29T11:31:38.113 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.39304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.37976.log 2024-03-29T11:31:38.113 INFO:teuthology.orchestra.run.smithi086.stderr: 86.2% -- replaced with /var/log/ceph/ceph-client.admin.39971.log.gz 2024-03-29T11:31:38.113 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-mon.c.log: 86.7% -- replaced with /var/log/ceph/ceph-client.admin.39304.log.gz 2024-03-29T11:31:38.113 INFO:teuthology.orchestra.run.smithi086.stderr:/var/log/ceph/ceph-client.admin.37976.log: 85.8% -- replaced with /var/log/ceph/ceph-client.admin.37976.log.gz 2024-03-29T11:31:38.147 INFO:teuthology.orchestra.run.smithi196.stderr: 92.2% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-03-29T11:31:49.279 INFO:teuthology.orchestra.run.smithi196.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-03-29T11:31:49.444 INFO:teuthology.orchestra.run.smithi086.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-03-29T11:31:49.807 INFO:teuthology.orchestra.run.smithi086.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-03-29T11:31:50.013 INFO:teuthology.orchestra.run.smithi196.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-03-29T11:31:50.086 INFO:teuthology.orchestra.run.smithi196.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-03-29T11:31:53.555 INFO:teuthology.orchestra.run.smithi086.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-03-29T11:31:53.615 INFO:teuthology.orchestra.run.smithi196.stderr: 94.7% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-03-29T11:31:53.723 INFO:teuthology.orchestra.run.smithi086.stderr: 94.6% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-03-29T11:32:27.516 INFO:teuthology.orchestra.run.smithi086.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-03-29T11:32:32.782 INFO:teuthology.orchestra.run.smithi086.stderr: 92.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-03-29T11:32:34.164 INFO:teuthology.orchestra.run.smithi086.stderr: 2024-03-29T11:32:34.164 INFO:teuthology.orchestra.run.smithi086.stderr:real 0m54.751s 2024-03-29T11:32:34.164 INFO:teuthology.orchestra.run.smithi086.stderr:user 2m32.705s 2024-03-29T11:32:34.164 INFO:teuthology.orchestra.run.smithi086.stderr:sys 0m5.556s 2024-03-29T11:32:49.448 INFO:teuthology.orchestra.run.smithi196.stderr: 92.5% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-03-29T11:32:49.450 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-03-29T11:32:49.450 INFO:teuthology.orchestra.run.smithi196.stderr:real 1m11.390s 2024-03-29T11:32:49.450 INFO:teuthology.orchestra.run.smithi196.stderr:user 1m57.517s 2024-03-29T11:32:49.450 INFO:teuthology.orchestra.run.smithi196.stderr:sys 0m4.414s 2024-03-29T11:32:49.450 INFO:tasks.ceph:Archiving logs... 2024-03-29T11:32:49.451 DEBUG:teuthology.misc:Transferring archived files from smithi086:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622694/remote/smithi086/log 2024-03-29T11:32:49.452 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:33:13.044 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/log/ceph to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622694/remote/smithi196/log 2024-03-29T11:33:13.045 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-29T11:33:32.151 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:33:32.357 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=6dff5224bb7c4eb7b905473dc818211e Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 1913, in task with contextutil.nested(*subtasks): File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 252, in ceph_log yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_1fb5d10608dccee73edd6ef8a2c7b804a097c324/qa/tasks/ceph.py", line 401, in create_rbd_pool teuthology.wait_until_osds_up( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 861, in wait_until_osds_up while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (91) after waiting for 540 seconds 2024-03-29T11:33:32.361 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-03-29T11:33:32.369 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-29T11:33:32.382 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-29T11:33:32.382 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:33:32.397 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-29T11:33:32.482 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-29T11:33:32.491 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-29T11:33:32.491 DEBUG:teuthology.orchestra.run.smithi086:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:33:32.494 DEBUG:teuthology.orchestra.run.smithi196:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-29T11:33:32.508 INFO:teuthology.orchestra.run.smithi086.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:33:32.511 INFO:teuthology.orchestra.run.smithi196.stderr:bash: line 1: ntpq: command not found 2024-03-29T11:33:32.514 INFO:teuthology.orchestra.run.smithi086.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:33:32.515 INFO:teuthology.orchestra.run.smithi086.stdout:=============================================================================== 2024-03-29T11:33:32.515 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 61 +467us[ +467us] +/- 92ms 2024-03-29T11:33:32.515 INFO:teuthology.orchestra.run.smithi086.stdout:^* hv02.front.sepia.ceph.com 3 6 377 62 -114us[ -125us] +/- 55ms 2024-03-29T11:33:32.515 INFO:teuthology.orchestra.run.smithi086.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 62 -259us[ -270us] +/- 86ms 2024-03-29T11:33:32.516 INFO:teuthology.orchestra.run.smithi086.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:33:32.517 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-29T11:33:32.518 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== 2024-03-29T11:33:32.518 INFO:teuthology.orchestra.run.smithi196.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 62 +486us[ +483us] +/- 92ms 2024-03-29T11:33:32.518 INFO:teuthology.orchestra.run.smithi196.stdout:^* hv02.front.sepia.ceph.com 3 6 377 59 -105us[ -107us] +/- 55ms 2024-03-29T11:33:32.518 INFO:teuthology.orchestra.run.smithi196.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 61 -249us[ -252us] +/- 87ms 2024-03-29T11:33:32.519 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-03-29T11:33:32.519 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-29T11:33:32.529 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-29T11:33:32.529 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-29T11:33:32.540 DEBUG:teuthology.orchestra.run.smithi086:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:33:32.626 DEBUG:teuthology.orchestra.run.smithi196:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-29T11:33:32.697 DEBUG:teuthology.orchestra.run.smithi086:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:33:32.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:33:32.729 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-29T11:33:32.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:33:32.761 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-29T11:33:32.770 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-29T11:33:32.777 INFO:teuthology.task.internal:Duration was 1301.643209 seconds 2024-03-29T11:33:32.778 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-29T11:33:32.787 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-29T11:33:32.787 DEBUG:teuthology.orchestra.run.smithi086:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:33:32.789 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-29T11:33:32.832 INFO:teuthology.orchestra.run.smithi086.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:33:32.846 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-29T11:33:33.146 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-29T11:33:33.147 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi086.front.sepia.ceph.com 2024-03-29T11:33:33.147 DEBUG:teuthology.orchestra.run.smithi086:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:33:33.167 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi196.front.sepia.ceph.com 2024-03-29T11:33:33.167 DEBUG:teuthology.orchestra.run.smithi196:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-29T11:33:33.188 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-29T11:33:33.189 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:33:33.210 DEBUG:teuthology.orchestra.run.smithi196:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-29T11:33:33.258 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-29T11:33:33.258 DEBUG:teuthology.orchestra.run.smithi086:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:33:33.280 DEBUG:teuthology.orchestra.run.smithi196:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-29T11:33:33.397 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-29T11:33:33.408 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-29T11:33:33.408 DEBUG:teuthology.orchestra.run.smithi086:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:33:33.435 DEBUG:teuthology.orchestra.run.smithi196:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-29T11:33:33.462 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-29T11:33:33.471 DEBUG:teuthology.orchestra.run.smithi086:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:33:33.477 DEBUG:teuthology.orchestra.run.smithi196:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:33:33.500 INFO:teuthology.orchestra.run.smithi086.stdout:kernel.core_pattern = core 2024-03-29T11:33:33.527 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = core 2024-03-29T11:33:33.550 DEBUG:teuthology.orchestra.run.smithi086:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:33:33.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:33:33.576 DEBUG:teuthology.orchestra.run.smithi196:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-29T11:33:33.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-29T11:33:33.604 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-29T11:33:33.614 INFO:teuthology.task.internal:Transferring archived files... 2024-03-29T11:33:33.615 DEBUG:teuthology.misc:Transferring archived files from smithi086:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622694/remote/smithi086 2024-03-29T11:33:33.616 DEBUG:teuthology.orchestra.run.smithi086:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:33:33.651 DEBUG:teuthology.misc:Transferring archived files from smithi196:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622694/remote/smithi196 2024-03-29T11:33:33.651 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-29T11:33:33.687 INFO:teuthology.task.internal:Removing archive directory... 2024-03-29T11:33:33.687 DEBUG:teuthology.orchestra.run.smithi086:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:33:33.690 DEBUG:teuthology.orchestra.run.smithi196:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-29T11:33:33.737 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-29T11:33:33.748 INFO:teuthology.task.internal:Not uploading archives. 2024-03-29T11:33:33.748 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-29T11:33:33.757 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-29T11:33:33.757 DEBUG:teuthology.orchestra.run.smithi086:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:33:33.759 DEBUG:teuthology.orchestra.run.smithi196:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-29T11:33:33.771 INFO:teuthology.orchestra.run.smithi086.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:33 /home/ubuntu/cephtest 2024-03-29T11:33:33.791 INFO:teuthology.orchestra.run.smithi196.stdout: 262166 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 29 11:33 /home/ubuntu/cephtest 2024-03-29T11:33:33.792 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-29T11:33:33.802 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-29T11:33:33.821 INFO:teuthology.nuke:Checking targets against current locks 2024-03-29T11:33:33.848 DEBUG:teuthology.nuke:shortname: smithi086 2024-03-29T11:33:33.848 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:33:33.868 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi086.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622694', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:09.230609', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKxPlMYyYmBRZDXnysowdOFSkkk1PV8lkGFIQWavm/+mvY2O+h5sZqC/0bFk9+9k39a/tF2jq129pkZF9EhNJGE='} 2024-03-29T11:33:33.872 DEBUG:teuthology.nuke:shortname: smithi196 2024-03-29T11:33:33.872 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-29T11:33:33.893 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi196.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/amathuri-2024-03-26_05:45:44-rados:verify-main-distro-default-smithi/7622694', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-29 11:02:09.229153', 'locked_by': 'scheduled_amathuri@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBITeA4FH29GBMbjGs2YpZjmD02Lke+TSUSNBt/bSNE5Jj2yN5L+e0jVZLnzu8Z5jXcWb02DMbALytQ3wv0SU21k='} 2024-03-29T11:33:33.909 INFO:teuthology.orchestra.console.smithi086:Power off 2024-03-29T11:33:33.910 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:33:33.936 DEBUG:teuthology.orchestra.console.smithi086:power off output: Chassis Power Control: Down/Off 2024-03-29T11:33:33.936 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:33:33.953 INFO:teuthology.orchestra.console.smithi196:Power off 2024-03-29T11:33:33.954 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-29T11:33:33.963 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2024-03-29T11:33:33.976 DEBUG:teuthology.orchestra.console.smithi196:power off output: Chassis Power Control: Down/Off 2024-03-29T11:33:33.976 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:33:34.001 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is on 2024-03-29T11:33:37.965 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:33:38.002 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:33:38.075 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2024-03-29T11:33:38.113 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is on 2024-03-29T11:33:42.078 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:33:42.113 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:33:42.190 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is on 2024-03-29T11:33:42.224 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is on 2024-03-29T11:33:46.195 DEBUG:teuthology.orchestra.console.smithi086:pexpect command: ipmitool -H smithi086.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:33:46.225 DEBUG:teuthology.orchestra.console.smithi196:pexpect command: ipmitool -H smithi196.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-29T11:33:46.306 DEBUG:teuthology.orchestra.console.smithi086:check power output: Chassis Power is off 2024-03-29T11:33:46.335 DEBUG:teuthology.orchestra.console.smithi196:check power output: Chassis Power is off 2024-03-29T11:33:46.408 INFO:teuthology.orchestra.console.smithi086:Power off completed 2024-03-29T11:33:46.437 INFO:teuthology.orchestra.console.smithi196:Power off completed 2024-03-29T11:33:46.541 INFO:teuthology.run:Summary data: description: rados:verify/{centos_latest ceph clusters/{fixed-2 openstack} d-thrash/none mon_election/classic msgr-failures/few msgr/async-v1only objectstore/bluestore-comp-snappy rados tasks/rados_api_tests validater/lockdep} duration: 1301.6432085037231 failure_reason: reached maximum tries (91) after waiting for 540 seconds flavor: default owner: scheduled_amathuri@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=6dff5224bb7c4eb7b905473dc818211e status: fail success: false 2024-03-29T11:33:46.542 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-29T11:33:46.622 INFO:teuthology.run:FAIL